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CHAPTER 1 


INTRODUCTION 


statement of Purpose 


Jt is the purpose of this paper to define the compctitive computer selection 
system used by the United States Navy. Other sclection systcms are described, and 


the Navy's system is evaluated in relation to these other systems. 


Organization of Study 


In kecping with its expressed purpose, this thesis is divided into several 
chapters, cach chapter provides the reader with a logical element from the total 
package. 

The coimpctitive computer sclection process used by the Depaitment ore 
Navy is highly regulated and owcs its origin to the fact that the Navy is a govern- 
mental agency with a central office responsible for the selection process. In 
@eapter Il, the variows conlrols exerted by Congress, the President, ~tne Departmen: 
@P Defense, and the Secretary of the Navy are cxaimnincd. The responsible central 


office is a very significant factor shaping the selection system. Accordingly, the 








mission and organizational relationships of this office are defined. 

Chapter HI examines the details of the selection procedures. The total cycle 
of acquiring a computer system involves two distinct and separate phases. First, 
the user conducts a system study to define the management data system and prove 
its soundness. ‘The second part concerns the procurement actions necessary to 
bring a computer system on board. This paper will address only the latter part of 
the cycle and thus assumes that a good system study has been conducted and the 
prospective user therefore knows his true systems requirements. Chapter II is 
therefore directed toward the five major activities that involve the central selection 
mittee. Ihe five activiiies are: Specification Review, Preparation of the Requestios 
Proposal and Selection Plan, Vendor Liaison and Validation of Proposals, Evaluation 
of Proposal, and Contract Negotiation and Award. 

Chapter JV discusses the results of a survey conducted by Dr. Norman 
Schneidewind. This survey purports to show four different catergories of users, and 
what practices are used, to select computer systems. In addition, the validation 
techniques of computer simulation and hand timing are discussed. The latter part 
of this chapter is directed toward two alternative evaluation systems--the Cost- 
Effectiveness and the Weighted Scoring methods. 

Chapter V evaluates each of the five major activities in the Navy selection 
process in relation to the methods described in Chapter IV. Chapter VI summarizes 


the findings of the study. 








Statement of Research Question 


me eee 








This thesis provides answers to the following questions: 

oO What is the relationship of the Navy's system to alternative systems that 
might be used? 

Q What are the Sicmpiicant@icaturcs of the campctitive computer seleemon 


used by the United States Navy? 


© What altemative methods could be uscd? 
° What are the strong points in the Navy's system? 
3) What are the controls imposed by higher authority? 


Oo Who in the Navy is responsible for the selection process? 





CHAPTER ff 


THE GOVERNMENT IEENVIRONMEENT 


The system the United States Navy uses to competitively select computers is 
a result of the interaction of many and diverse influences. There are strong 
pressurcs exerted on the Navy by such external forces as the United States Congress, 
ane President of the oh states, the Bureau of the Budget, the Depanimenmmen 
Defense and many others. Jn addition, there are active forces within the Navy 
itself. The most significant action clement in the Navy is the Automatic Data 
Processing Equipment Sclection Office. The purpose of this chapter will be to 


define these influences. The effect of these forces will be examined in Chapter III. 


Congressional Influence 


The most significant influence exerted by the Coneress is Public Law 89-306 
enacted on October 30, 1965. ‘This law is known as the Brooks Bill because the 
legislation is a product of the Government Activities Subcommittee of the House 


Committee on Government Operations, Congressman Jack Brooks is chairman 





: : les : : : ; ; ; 
of this subcommittee.” The need for this legislation was defined by the subcommittee 


in its report on H.R. 4845: 


Since 1958, up to the time of the hearings on H.R. 4845, the GAO has issued 
approximately 100 audit reports to agencies, congressional committees, and to 
Congress revealing serious shortcomings in the manner in which specific agencies 


acquired and/or utilized ADP equipment. The major deficiencies cited in these 
meports have been: 


(a) Inadequate feasibility studies; 

(b) Uneconomical and ineffective equipment utilization; 

(c) Overpayments resulting from inadequate management practices; and 
(d) Uneconomical procurement of equipment. 


Based on this need, -the Brooks Bill purports to satisfy three vital management 
needs: (1) more adequate management information, (2) optimum utilization of ADP 
(Automatic Data Processing) equipment through sharing, and (3) multiple use and 
economic acquisition. The subcommittee indicated that an economical acquisition 
program involved three principal factors: 

(a) Improving the Government's bargaining position through volume acquisition; 

(b) Basing lease versus purchase evaluations, whenever possible, on the long- 


term value of the equipment to Government as a whole; and 


(c) Selecting that equipment for purchase which, on a Government-wide basis 
Offers the largest purchase advantage. 4 


Ly, S., Congress, House, Committee on Government Operations, Automatic 
Data Processing Equipment, Hearings, before a subcommittee of the Committec on 
Government Operations, House of Representatives, on H.R. 4845, 89th Cong., lst 
eeas,, 1965, p. II. 


rR. Rev Geo ro Wine Olam cieceas alo OO, Iwi. 


Sibid., p. 23. 








To satisfy the needs of management as cited above, Public Law 89- 306 
increased the authority and responsibility of three federal agencies: the General 
Services Administration, the Bureau of the Budget, and the National Bureau of 
Standards in the Department of Commerce. 

The Bureau of the Budget is charged with the fiscal and policy control of ADP 
management of the government. The bill provides that the actions of any agency 
taken under the authority of the law are subject to the review and approval of the 
Bureau of the Budget. 

The Administrator of the General Services Administration (GSA) is charged 
with the primary responsibility for coordinating the government's ADP management, 
subject to the policy and fiscal control of the Burcau of the Budgct. ‘The law 
specifically directs the administrator to “coordinate and provide for the economic 
and efficient purchase, lease, and maintenance of automatic data proccssing 
equipment by Federal agencies."4 The bill however docs limit this authority. Each 
agency was given the latitude to continue determining what their ADP requirements 
Were. 

The National Bureau of Standards is Reena for the technical aspects of 
this coordinated management program. The Bureau is responsible for conducting 


rescarch and development programs based on the needs of the government. The 


4Vederal Property and Administrative Services Act of 1949 (63 Stat. 377) as 
amended, Sec. 111.(a). 

















drafters of the bill intended for the Bureau of Standards to direct its attention toward 
the problem of lack of equipment compatability. The intent of the law however was 
not to authorize the Bureau to launch a broad rescarch and development program. 
Rather, the intent was aimed at supp!ementing the government research effort in 
coordination with other federal agencies and monitoring developments in the industry 
affecting areas covered by the law. A copy of Public Law 89-306 is shown in 


Appendix A. 


Presidential Influcnce 


President Lyndon B. Johnson on June 28, 1966, sent a memo to the heads of 
all federal agencies. The purpose of the memo was to stimulate action to improve 
the management of the ADP resources in the government. The President referred 
to the recognition that the Congress had given to this problem by enacting the Brooks 
Bill. President Johnson pointed to the fact that the federal government was using 
2,600 computers, employing 71,000 people in the field, and spending over $2 billion 
annually to acquire and operate ADP equipment. The President said: “I want the 
head of every Federal agency to explore and apply all possible means to use the 
electronic computer to do a better job . . . manage computer activity at the lowest 
possible cost." The President pointed to four specific areas that were to be 


— 


| S Lyndon B. Johnson, Memorandum to Heads of All Federal Agencies, 28 June 
OO). 








given priority attention: better methods of precuring ADP systems; achieving 

better utilization of existing capabilitics through sharing and joint usc arrangements; 

re-vtilizing excess eduipment when possible; and achicving greater compatability. 
The President indicated to the Heads of the Agencies that he expected them to 

cooperate fully with the Bureau of the Budget, the General Services Administration, 

and tne Department of Commerce in accomplishing these actions. (See Appendix B 


for a copy of the President's meinorandum.) 


secretary of Defcnse Influence 


secretary McNamara acted upon the Pie sident's memorandum with a statement 
of policy to the top management of the Department of Defense (DOD). © The secretary 
indicated his desire to attain the objectives set down by the President but further a 
desire to set an example to the rest of the government agencies. The memorandum 
pointed to the fact that DOD operated 2,000 of the 2,600 computers cited in the 
President's memo and employed 51,000 of the 71,000 federal employees engaged in 


the ADP cffort. It was therefore the view of the Secretary of Defense that DOD had 


Proper S. McNamara, Memorandum to The Secretaries of the Military 
Departments, The Chairman of the Joint Chiefs of Staff, The Director of Defense 
Research and Enpinecring, The Assistant Secretaries of Defense, The Director of 
Defense Agencies, 29 July 1966. 
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to bear a large share of responsibility for improved manageinent of the govemments 
ADP resources. 

The Secretary's memorandum defined the areas that required management 
attention. There are several items of specific interests in this paper: 


(a) We must, prior to computer selection, develop and issue systems 
specifications which adequately describe the systems to be performed and which 
will resuit in selection of computers which can satisfy the requirements of that 
specification. We should not be forced to acquire additional units at later dates 
or replace the computers prematurely because of selections based on inadequate 
system specifications or on less than responsive vendor proposals. Military 
Departments and DOD agencies should centrally prescribe and control the 
development of systems specifications for computer acquisition purposes. 


(b) We must make competitive computer evaluation and selection a 
professional endeavor and organize and staff for it accordingly. Staffs performing 
this function for the Senior automatic data processing policy officials should be 
divorced {vom computer-using organizations. Steps should be taken to assure 
that these staffs are the sole point of contact with the computer industry on all 
matters pertaining to computer selection for their respective agencies. 


(c) ... Further, computers will not be selected until the performance of the 
complete pee) software » Package a in the ie pevueee and 


a a Eh a ES SS GS 


ane anata test. (Empha sis sanlsiedl y/ 


secretary of the Navy Action 


The Secretary of the Navy initiated a study effort within the Department of the 
Navy to define the actions that should be taken to satisfy the policy statements of 


President Johnson and Sccretary McNamara. 


"ibid. 
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An instruction signed by Secretary of the Navy Paul H. Nitze on June 30, 1967 
established the Automatic Data Processing Equipment Selection Office (ADPESO). 
The mission assigned to this office is: 

a. To evaluate and seiect for approval by the Special Assistant to the 

Secretary of the Navy (SASN), automatic data processing eQuipment to be 


acquired by the Department of the Navy, except as defined in paragraph 4 
beiow. 


b. To act as the Department of the Navy contracting office for the 
procurement of ADVE. 


The exception mentioned above relates to equipments integrated with weapons 

systems, punch card equipment, and other special situations exempted by SASN, 

The functions assigned ADPESO are: 

(a) Serve ag the principal point of contact with the industry on matters 
concerning validation, evaluation, selection, and procurement of ADPE for the 
Department of the Navy. 

(b) Develop and administer for the Department of the Navy plans, policies, 
procedures and methods governing ADPE specification preparation, proposal 
solicitation, proposal evaluation and selection. 

(c) To solicit sources, negotiate and award contracts, place delivery orders 
acainst Federal Supply Schedules and administer contracts and orders for such 
ADPE acquisitions. 

The Director of the Automatic Data Processing Fquiprnent Selection Office, 

reports to the Special Assistant to the Secretary of the Navy. Thus he is very high 


pp eemee e  e 


8U.S., Department of the Navy, "Automatic Data Processing Equipment 
Selection Office; establishment of," Secretary of the Navy Instruction 5430. 81, 
Department of the Navy, Washineton, D.C. , 20390, June 30; 1967; 








iat 


in the Navy department hierarchy and is separated from computer-using oreanizatious. 


(See Appendix D for a diagram of this organizational alignment.) 


ADPESO Action 


Upon completion of the first milestone of organizing and staffing, management 
turned and directed it attention toward the mission assigned to ADPESO. In order for 
this to be done, the methods of operation had to be defined and approved. This effort 
required ADPESO peuconnel to gather the various directives and regulations that 
apply to ADP management jn the government. There is a large number of directives 
toward this end. (See Appendix E.) For ae purposes of this paper however, only 


the most significant ones will be discussed. 


Bureau of the Budget Circular No. A-54: 


Policies on Selection and Acquisition of Automatic Data Processing Equipment 


The purpose of this directive is to prescribe the government's policy 
concerning ADP equipment selection and provide guidance relative to the method of 
acquisition-lease, purchase or a lease with an option to purchase method. 

The government's policy concerning selection specifies: that all selections 
will be made based on a set of systems specifications; that selection procedures will 
assure equal opportunity for all manufacturers to participate; that the two prime 


factors to be considered in the selection process are (1) the equipment's ability to 








fulfill the systems specification, and (2) its overall costs including those applicable 
to acquisition, preparation for use and operation. 

The agency in question is obliged to select that method of ADPE acquisition 
Which offers the greatest advantage to the government. The purchase method is to 
be used when all the following conditions exist: 

a) There is a reasonable expectancy that the ADPE in question can be 

successfully and advantageously used; 

© That after a comparative cost analysis has been conducted, it is 

established that a cost advantage can be obtained in six years or less 
and the cost of money? has to be used in conducting this cost analysis; 

e The ADPE will be able to satisfy the requirements of the system beyond 

the point where the purchase method provides a cost advantage; and 

© The feasibility and economics of performing cquipment maintenance with 

in-house resources has been considered. 

gine lease-with- option-to-purchase method is to be used when It 1s m#eceea a, 
or advantageous to acquire the equipment, but it is preferable to defer temporarily 
the decision to purchase when circumstances do not fully satisfy the conditions 


which would indicate purchase. 


The procedure used is defined by Bureau of the Budget Circular A-94. A 
sample problem from this circular is shown as Appendix J. 








1) 


The lease method, without option to purchase, should be used when it is 
advantageous or necessary to acquire the equipment and when it has been determined 


that anyone of the prerequisite conditions of purchase cannot be attained. 


Department of Defense Directive 4105. 62: 
Proposal Evaluation and Source Selection 

The purpose of this directive is to establish the objectives, principles, and 
policy for the evaluation of proposals and the selection of contractual sources. The 
ss cctive delines three very Sienificant parties and their roles in the selection 
TOCcess. 

First is the Source Selection Authority (SSA). He is an official designated to 
Gimect the source selection process and to make the source selection decision. Wie 
principal duties of this official are: 

1. Designate the chairmen and the membership of the Source Selection 
Advisory Council. 

2. Establish guidelines for the actions to be taken by the Source Seltecmien 
Advisory Council and the Source Selection Evaluation Board. 

3. Make the decision as to the source, after an in-depth review and 
consideration of all information and data available from the Source Selection 


Advisory Council and the Source Selection Evaluation Board. 








14 


Second is the Source Selection Advisory Council (SSAC). It is composed of a 
group of senior military and civilian personnel and is appointed by thc SSA and acts 
as his staff and advisors in the source selection process. ‘The principal duties of 
this group are: 

J), Establish the evaluation criteria and assign weight factors to each item 
using the recommendations of the Selection Plan which was prepared by the System/ 
Heoject office, in this case ADPizS0. 

2. Designate the chairman and members of the Source Selection Evaluation 
Board. 

3. Review the findings of the Source Selection Evaluation Board, 

Third is the Source Selection Evaluation Board (SSEB). This is a group of 
military and civilian personnel, from the various functional and technical areas, 
mie duties of this group are: 

}. Evaluate and score the proposals submitted by the bidders; and 

2. Prepare a narrative justification for the cvaluation results. 

This dircctive thus establishes a three-level review process for the cvaluation 
of system proposals. 

Based on these various directives and the knowledge and expertise of the staff 
of ADPESO, the Navy's computer selection process was developed. While the many 
directives provided broad policy guidelines, detail methods or techniques were not 


specified. 








is 


The method of evaluating the proposals is based on a system called the 
“Cost- Value Technique" from a book by Edward O. Joslin, Computer Selection. 10 
Mr. Joslin wrote this book while attending Boston College and working for DP 
mamipment (Sclection) Office of the U.S. Air Force. Currently he is Head of the 
Techniques Development and Analysis Division at ADPESO. 

The next chapter will treat the details of the Navy's Competitive Computer 


Ssclection System. 


LOR dwvard O. Joslin, Computer Selection, (Reading, Massachusetts: Addison 


Weslcy Publishing Company, Inc., 1968), p. 20. 








CHAPTER is 


U.S. NAVY COMPETITIVE COMPUTER SELECTION SYSTEM 


The Navy's competitive computer selection system starts when the system 
specifications for a particular data system are sent to the ADPESO for action. The 
actions taken by ADPJ}:SO can be arranged into five steps: 

Specification Review, 

Preparation of the Request for Proposal (RFP), and Selection Pian, 
Vendor Liaison and Proposal Validation, 

Proposal Evaluation, and 

Negotiation and Contract Award. 


Each of these areas will be discussed and explained. 


Svecification Review 


The ADPESO actually performs three different reviews of cach set of 
Specifications. The purpose of the first is to determinc if the user's requirement 
can be satisfactorily satisfied by sharing a computer system currently operative in 
some other area, or by reutilization of some system not in use or scheduled to 


become idle and owned by the government. The second review is conducted to 
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by 


determine whether this requirement should be satisfied by a competitive or a sole 
source procurement action. While these two reviews are important they are not 
gcrmanc to the subject of this paper; accordingly attention will be devoted to the 
third review only. 

The third review performed by ADPESO is concesmiedmwith themtatascontaiied in 
iiie system specifications. 1 Three parts of the user's requirements are analyzed. 
First, the justifications provided for conditions stated as mandatory are checked. 
Second, the review examines the rationale provided for the requirements described 
as desirable features. The last part of this review is concerned with the workload 


description. 


Justification of Mandatory Requirements 


A mandatory requirement is defined as, “those requirements which the supplicr 
must satisfy to be considered. "2 The Navy's policy is that the mandatory requirements 
shall not be stated in a manner that will restrict the free competition among all of 
gee gualified bidders. Purther the requirements cannot be oriented toward any 


specific vendor's cquipiment, software, or scerviccs. 


Une , Department of the Navy, Specification, Selection, and Acquisition of 
Automatic Data Processing Equipment, Secretary of the Navy Instruction 10462. 13, 
Department of the Navy, Washington, D.C. 20390, p. B-2. The format for thie 
systems specifications is shown in Appendix F. 


2thid., p. B- 11. 
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The specifications must be reviewed to mnsume that the user hase giied 
all mandatory requirements, and that the number of them is held to a minimum. In 
addition the reviewer must insure that the requirement does not restrict free 
compctition in any way. The justification for any mandatory item has to be expressed 
in terms of ". . . why alternative approaches would not prove satisfactory. This 
justification should show that the cost of effectively satisfying the requirement in 
any alternate way would be prohibitive, or that the required effectiveness can only 
be satisfied in the way shown. "3 A special check is also made to ensure that 
mandatory supplicr capabilities, i.e., system maintenance, training, and conversion, 
ctc., are not confused with mandatory system requirements. An example of some 
mandatory requirements and the types of justification required is shown in Figure 1. 

Following are examples of requirements that could, if properly justified, be 
mandatory. 4 

Oo Equipment Availability 

If the equipment must be installed and operational by a specific calendar 

date to meet the requirement of the system(s), the user indicates the latest acceptable 
operational date. It should be noted ihat to speciiy a date is an Cxcepligmevemiie 


moral practice. 


SIbid., p. B-2. 
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Muginol bens) Requirements 
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a. Supplier Support - 
Hist agvec to. 


The supplier 


(1) Train or provide training for, 
system analyst, programmers, 
and operating personnel. 


(2) The maintenance personnel 
provided must have a SECRET 
Securiy Cleat ane 


b. Workload Demonstzvation - The 
benchmarks must be successfully 
min in 5.00 hours or less. 


e. COBOL Compiler - The COBOL 
Compiler proposed and demonstrated 
with this system must be a USASI 
Standard COEOL Compiler (X3. 23- 
1966). 


"meas at MRO ty Eire tap Crete ne! LBS RIG OTE © OEE Del etlins Tal OT teh 
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Instruction J0462. 
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SOC ale oe Department of the Navy, Specification, 
Acquisition of zAUOIEELE CS Data mOLESSINE eguipne nts 
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Justification 








(1) The supplier must at least be 
responsibie fox the training to 
assure thatthe training 16 er the 
proper type. If this trainine does 
not provide mere ner injomiiavign 
the system cannot be properly 
utilized. 


(2) This is necessary to meet local 
Security requinements. sce 
paragraph ss 

Completion of the benchmarks is 

required to demonstrate the timing 

capability of the proposed system. 

Failure to complete the benchmarks 

in less than 5.00 hours would indicate 

that the supplier's equipment could not 

complete the activity’s projected work- 
load in the 600 hours a month available. 

Completion of the benchmarks will a 

also provide the validation of the 

existence of a suitable executive system 
and compiler. 


This is required for compliance with 
Department of Defense acceptance of 
the USASI Standard. 


ke tm ihe 2 ta ccm ee om ene ec at ee pete ee 7 maa Tee Rea 


1, Selection, and 
SCCretary Of ume ae 














Figure 1 —- Mandatory Requirements and Their Justification 
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o Special Input/Output Requirements 
If special or nonstandard equipment inust interface with the required 
equipment, or if some special charactcristics are required, such as special 
@eoaeacter scts for the printer, minimum print characters per linc, eic., the 
requirement must be stated in such detail that the vendors will understand it. 
© Operational Requirements 
The user would deme liere any ,cquirement that existe lei amescm7e 
for planned expansion, mobilization, and unavoidable workload peaking. Additionally, 
any requirement that all or certain portions of processing must be accomplished 
within a certain time frame are to be indicated. If the system is a rcal-time system 
the user can indicate the response time required. If the user has a peak load that 
must be completed during a specific time frame, the requirement must be stated 
in such sufficient detail that the suppliers can properly evaluate the situation. 
© Commiunicatious 
The user may have special communications requirements, such as 
remote stations, interface with communications network terminals or tributary 
terminal devices, ctc. If this is the case then the requirements must be defined so 
the vendors can respond properly. 
° Security Requirement 
This is particularly applicable to Department of Defense aciiviluacs. lf 


the user's system docs involve national security matters the rules applicable to the 
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supplier and the proposed installations must be stated. Jn addition the epplicable 
rerulation must be cited. 

© Expansion Capability 

The user should state the expansion capability required of the system for 
each year of the systems life. This is specified as a percentage increase over the 
first year, or by increased ycarly extension factors. 

There are other mandatory conditions which might be specified. Normally the 
user will state the amount of time, per month, that is acceptable for compiction of 
the workload. This is determined based on the time required to complete the 
benchmark. The determination process will be reviewed later. 

The experience to date indicates that most conditions claimed as mandatory 
are not usually sufficiently justified, so the requirement is classified as a desirable. 
It is also possible that as the threshold or limit changes, the condition can be 
treated as a mandatory requirement under one set of criteria and alsoasa 
desirable feature using a different criterion; for example, it might be a mandatory 
requirement that the supplier install the proposed equipment not later than a given 


date, and desirable for it to be installed earlier. . 


s) Ne ; ae 
E.O. Joslin, "Competitive Computer Selection Within the Department of the 


Navy,” (unpublished paper presented to the Diebold Group, Inc., October, 1969. 
Mr. Joslin is Head, Techniques Development and Analysis Division of ADPESO). 
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Rationale For Desirable Features 


A desirable feature is defined as ‘those features, characteristics, and 
services which the user desires, but which are not mandatory for a supplier to 
provide to be considered as meeting the overall requirement." 

For each desirable feature specified by the user, a statement of value has to be 
given as a justification. The value has to be stated in terms of dollars over the life 
of the system. However, the rationale required to define a desirable condition is less 
rigid than that required 5 a mandatory item. A review of statements of desirable 
features, will attempt to find two characteristics. First: is the item adequately 
described and second does the rationale Aen why the item is desirable. The only 
acceptable explanation of desirability is one expressed in dollar terms. In fact when 
possible the user is suppose to provide two value or worth figures. One figure 
defines the savings or cost avoidance that will be realized if the feature is provided. 
The second figure provides the cost of achieving this desirable in any possible 
alternative way. The lower of these two figures is the value assigned to that feature. 

Following is a list of the type items which may be listed as desirable features. / 

© Equipment Availability 

The user can indicate the desirability of having the equipment installed 
as much as three months in advance of the required delivery date. 

6SecNavinst 10462. 13, p. B-1l1. 
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O Software 
Jf the user desires any software packages to be provided by the vendor, 
each one is described. Soime exaniples are: report gemenarons, sort 1 oul = 
scientific subroutines (such as general factor analyses, multi- discriminant analyses, 
etc.), and PERT cost and time. 
© Supplier Support 
The user can specify items of support desired of the supplier such as, 
training to be conducted on- site, some number of man-months of analyst support in 
conversion of programs to the new system, and on-site maintenance. 
re) Mathematical Computation 
Special mathematical capability can be requested such as the ability to 
handle floating point computation. 
6 Utilization Log 
The user can request that the equipment, in conjunction with software, 
provide a daily utilization report by job number. 
Figure 2 shows some examples of desirable features and the corresponding 


value statements. 


Statement of Workload 


During the review of the mandatory and desirable features, the 


reviewer must also concern himself with the workload definition. This is so 











i, 


Blow charting 
prograin for 


COLOL 


Expansion 
capability of up to 
mity per Cent over 
tiie stated life 
workload 





It is estimated that a potential cost avoidance of $60,000 
would accrue to the government with the use of a Flow 
chart preparation prograra. This is based upon 
programimers devoting 5 per cent of thcir time to the 
task of flow charting. A siaff of thirty earning an 
average of $9,000 annually for four years is $1,080,000. 
Five per cent of this figure is $54,000. However, it is 
estimated that this would cost 528 hours of computer 
time over four years. At $37.50 per hour (non-peak 
time valuc), this is $19,800 total cost. Therefore, a net 
savings of $34,200 will result. An alternate source of 
supply for such a program indicates a cost of $8,000. 
This cost wes obtained from quotes received from 
private industry suppliers. Therefore, this altexnate 
source cost of $8,000 is taken to be the value of this 
program. 


A system which would require the full five hours to meet 
the mandatory requirement for completing the 
benchmarks would have the required life of five years 
and is expected to cost $1,000,000. However, the stated 
system life is based on expansion of the workload which 
has already been assigned to this organization. In 
addition to the capability to handle the known workload, 
it would be desirable to be able to handle two additional 
projects which have a fifty per cent chance of being 
assioned to this organization. The first project (project 
XYZ) would require an additional capability of approxi- 
mately thirty per cent of that required for the stated 
workload. The second project (project ABC) would 
require an additional time capability of twenty per cent 
and would also require the availability of a rermote tem 
terminal. Our ability to handle project XYZ could save 
an ultimate expenditure of $300,000 over a three year 
period in contracted computer time. Likewise, the 
ability to handle project ABC could result in savings of 


Pigure 2 Slaremente a: Value 
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Desitable Features Value Statement 


p200, 000. However, because there is only a fifty per 
cent chance of these projects being assigned to this 
organization only half of the possible $500,000 saving, or} 
9200,000 is being stated as the value of this expansion 
capability. 















3. supplier provided 
on~ site training of 
sixty 
programmers, 
thirty analysts, 
and ten operators 


If this training is not provided by the supplier, a cost of 
approximately $20,000 would be incurred in hiring a 
consultant to supply this training in accordance with the | 
supplier's direction. If this training is not accomplished | 
on-site, an additional cost of approximately 200 man- | 
weeks (assuming two week courses) of per diem or 

$19, 200 plus travel cost could be incurred. 


If on-call maintenance is all that is offered, this would 
mean one or two hours of additional lost time per call, 
and with the expected five calls a month, this could mean 
ten hours of lost time per month or 600 hours over the 
life of the system. ‘his lost time 1s estimatedjiia 
worth $12,000. 


4. On-site 
maintenance 


he I a OE ORE ipa ope eee ot 


SOURCE: U.s., Department of the Navy, Specification, selection, and 
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Acquisition of Automatic Data Processing Equipment, Secretary of the Navy 
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because many of the mandatory and desirable features relate to the vendor's 
capability to complete the workload satisfactorily. 

The review of the using activities workload is concerned with insuring that the 
expected workload is described in terms of the expected levels of workload and in 


terms of the representative programs or tasks plus the related extension factors 
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applicable to the workload levels. Each of these factors, workload levels and 


representative programs, will be discussed individually. 


Expected Workload Levels 


The workload to be processed on any system can be considered as consisting of 
a series of various levels. As the system ages, the workload usually increases 
from one level to the next. However, because of the problems associated with 
projecting future workload erowth, it would be unreasonable to expect that the user 
could be 100 per cent accurate in projecting the workload for the life of the system. 
Accordingly the user is directed to use Prpeabances to predict future workloads in 
the following manner:® 

l. System Life - the user prepares a chart that shows the number of years 


the system will be in existence. 


a. SYSTEM LIFE 





lst Year | 2nd Year 3rd Year 4th Year 5th Year 


Sy oslin, YCompeticive COmpule: Sereciion, py lo: 








2 °rojected Growth - an approximation of the predicted growth in 


workload is superimposed on the basic "System Life" chart a.. The vertical axis 


represents workload in hours-per-month. 


b. PROJECTED GROWTH 


Workload 





Ist Year | 2nd Year 3rd Year 4th Year ! Sth Year 


o Workload Levels - the user constructs a workload level line by drawing 


a horizontal line through the midpoint of the projected growth line for each year. 


©. WORKLOAD LEVELS 





TS Level 5 
cv, Grow Level 3 
St eae 4 ected Me 

© eo lbevou? 
~- 

SSE | Level 
lst Year 2nd Year 3rd Year 
4, 


Level of Provability - for each year a probability factor is assigned to 
each workload level. The total probability for any one year is considered to be 100. 
In illustration d. the user has indicated that during the first year there is a 90 per 


eent chance that the workload will be at Level i and a 10 per Cent chance it will be at 








Workload 
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Level 2. Note in the illustration below that the number of workload levels was 
increased to seven. This was acne to take care of a siluation where inere 14.4 
probability the workload will exceed the projected growth line in the fourth and fifth 


Nears. 


fee VEL OF PROBABILITY 









0 2 5 Level 7 
0 5 1S Level 6 
0 15 70 Level 5 
10 75 Level 4 
so.” 5 0 Level 3 
5 0 a 0 Level 2 
0 0 0 Level 1 
| a 
isia\.ecar Ze) 3 Cat ord Year | 4th Year Diltey Can 


Representative Programs 


This is a very important part of the systems specifications. ‘The representative 
programs are the vehicles that will be used to time the proposed computer systems. 
The essence of this approach is that a program called a "representative program” is 
selected to represent a specific portion of the total monthly workload. Thus, a group 
of representative programs represents the total monthly workload. These programs 
which neon benchmarks, in conjunction with the extension factors, provide the tools 
to project the monthly running time for the systems proposed by cach vendor. 

In view of the importance of this factor the review conducted by ADPESO is 
particularly critical. The reviewer looks to insure that certain facets are properly 


addressed. For cxample, the benchmarks should be written in a standard higher 








Zo 


level programming language such as: USASI Fortran or USASI COBOL. The vendor 
should be able to run all programs during a single half-day benchmark demonstration. 

The reviewer has to insure that the user has selected a truly representative 
program. This means that the benchmark problem must be representative of the 
following:? 

1. Types of processing: logic, computation, housekeeping, etc. 

Time requirements: compile, execute, tapes, card wnits, printers ere: 
3. Equipment Requirements: core, input/output channels, tapes, card units, 


Printers etc. 
4. Order of problems (priority). 


No 


The review also has to establish that the user has properly equated each 
benchmark to a portion of the workload. This is particularly important for as 
previously mentioned, the performance demonstrated on the benchmark is extra- 
polated to define the total monthly workload. 

The extension factor, a number derived for each representative program, is 
Eee next vital clement to be verified. The running time of each benchmark is 
multiplied by this factor to arrive at the monthly running time for the class of tasks 
represented by that program. An extension is required for both sequential systems 
and for multi-processing systems. The user should have prepared a table similar 
to Figure 3.. For each representative program there should be entries in the table 
showing the total monthly time consumed by each program, the running time of the 


representative task or program and the division of the total by the representative 


 Bdward ©. Joslin, Computer Selection (Reading, Massachusetts: Addison- 
Wesley Publishing Company, 1968), p. 77. 
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Sort MP geh aa oe 145.00 0.45 322 
y3-1A We io i sunte 125.00 QO. 25 500 
Card leaden 11.5200 0.03 3833 
dit otal thruput IPAQ ON 02 t 160 
0-42 Mag. tape 80.00 0.60 | ce 
Card Reader 20.00 G50 4.0 
Printer ILO 101 Oe25 409 
Update Tomar iene 100.00 O26 G75 
eae) Mag. tape 70.00 0.10 700 
Card Reader ZonUe 0.05 000 
Pyne, 50.00 RG 500 
Petar x 1 Total thruput 90.00 0.45 200 
Inversion Cargei.caccr Ono 0.02 175 
im 6A Mag. drum 24.00 Bee 160 
Printer oo OAod 150 
FORTRAN ‘Yotal thruput 85.00 0.17 900 
Compile Mag. tape e200 CS: 520 
H-3a .Card Reader 6.00 0.02 300 
Printer 4,00 O70 400 
COBOL Total thruput 40.00 0.12, sae 
Compile Mag, tape 38.00 Conk B45 
G2 - Carcdueader 4.00 0.04 100 
Printer SOU 0.04 Cs 
ape to Tolal thruput 300.00 100 300 
mvint Mag, tape 300.00 1.00 300 
-4 Printer 300.00 TOG, 300. 
JOM ON ec. 
PAGE > Pearce. ee 880.00 


ee ee er ee eee —_—=— 


SOURCE: Eup ccm , Department of the Navy, Specification, Selection, and 
Acquisition of AGtomente Data Processing Equipment, Secretary of the Navy _ 
Instruction 104 62. 13,-Bepartment of the Navy, Washington, D.C. 20390, p. -B-4. 




















Figure 3 -—~ Representative Freerams 
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program and this becomes the extension factor. Expressed another way, the 


extension factor is: 


Therefore: 


X( Zp) 


Ny << 


Total monthly time to perform total task set on 
present cquipmert system 


Throughput time to run the representative 
program 
(1) on present equipment (subscript a) 


(2) on proposed equipment (subscript b) 


Extension factor for representative task. 


Supplicrs time for total task10 


The user also has to show how these individual representative programs can 


be grouped into a mix of programs that has its own extension factor, which is 


representative of the total workload. The purpose of this mix of representative 


programs and related factors is for use in demonsirating the multiprogramming 


or multiprocessing capability. The mix of programs as modified by quantity and 


provisions, should be such that when the times for the present representative 


programs are multiplied by the extension factors the result will be the total time 


which should closely approximate the present total time. Figure 4 demonstrates 


this situation. 


lOsecNavinst 10462. 13, p. B-5. 
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“This extension factor for the mix is derived by examining the information contained 

in Figure 4 and obtaining the lowest practical extension factor to reduce the number 

of problems to be run in the mix while retaining the required representative nature of 

the mix of problems, which in this case is 160. This extension factor is then 

divided into each of the sequential extension factors to obtain the quantity column. 

The provisions columin is then used to make the input/output total time when extended 

by the mix extension factor cqual to the total projected input/output time. This mix 

of tasks (15) can then be used as a proper demonstration of a supplier's multi- 

programming or multi-processing. ha ee 
~ SOURCE: WE a Department of the i pacd Epecification, peleear and 
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iction 10462. 13, Tg anaanICuTl of the ae Washington, D.C. 2-300 tiem 
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The extension factors initially calculated are for the first year of the systems 
life. Accordingly, the user must also provide extension factors applicabie to the 
second, third, fourth, and fifth year of the systems life. 


The final check the reviewer makes is to insure that the user has provided a 
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description of each of the benchmark programs and copies of the program detail 
listings. 

If the user has properly justified the mandatory requirements, evaluated the 
desirabie features, and described the worlJoad, the specifications are ready for the 
next action. The specifications are forwarded to the Source Sekitton Evaluation 
Board for the preparation of the Request for Proposal which will be forwarded to the 
vendors, and the Selection Plan which is held in-house for official use only. 


s 


Preparation of the RFP and Selection Plan 


re a gE pte, ee a a 


Provided the specifications were properly prepared, the preparation of the RFP 
and Selection Plan is a relatively simple task, 11 

The RFP consists of a cover letter, the systems specifications, less the 
justification of mandatory requirements and the rationale for desirable features, and 
a set of instructions addressed to the vendors. ‘These instructions tell the vendors 
how to prepare their proposals; lists some mandatory administrative details and 
defines how the cost for the proposals should be computed and shown. Before this 
can be mailed to the vendors through, the Selection Pian has to be developed. 

The purpose of the Selection Plan is to establish a detail set of iustructions that 


will govern how the vendor proposals will be evaluated. It should be noted at this 


point that in the Navy's system it is the desirable features requested by the user that 


Wh oslin, Gormpctivive Computer mercemouw p27. 
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are evaluated. The mandatory requirements are not evaluated they are validated. 12 

The desirable features and the vaJues assigned to each are carefully analyzed. 
Based on this information a value templet is constructed for each feature. These 
templets show how the total value for each item will be apportioned as various 
quantities or qualitics of the item are proposed. Jn actual practice these value 
statements are prepared when the RFP is being prepared. The purpose of this is to 
insure agreement between the RFP and the Selection Plan. 

Another cxamination of the mandatory requirements is also conducted at this 


a 


time. ‘The reason for this second review or these requirements is to determine if 
any items initially classified as mandatery can be re-classified as desirable. 
Experience has shown that some mandatory items are really desirable in the sens 
that dollars can sometimes make up for a slight deliciency in a mandatory 
mequirement. 13 

There are four parts in a value statement: 

i A Statement of the Desirable Feature - provides the exact and entire 
statement of the desirable feature as it appears in the RFP. 

Jap Rationale - This section shows the derivation of the worth or value of 
the feature. The worth is defined by considering the following: 


ale The cost of doing without the capability; 


lor The cost of satisfying the requirement by using in-house resources; 


sececNavInst 10462.13, p. B-1l. 


13; oslin, "Competitive Computer Sclection,"” p. 23. 
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e. ‘Yhe cost of having the capability provided by some contractor other 
than the one bidding on the system (must always be preceded by a. or b.); 

ae A cost less than any of the above which would reflect what the 
activity would apy to get this capability (must be preceded by a., b., or c.); and 

oe The worth of each feature must be caiculated for each year of the 
systems life. 
The lowest worth derived above becomes the value of the feature. 

3. Rempicu= tells how the worth or value derived is to be distributed. 

4, Valuation ~ is actually done on a separate standard form which is 
attached to the value statement sheet. (A sarnple value statement is shown in 
Fieure 5.) 

The Selection Plan also defines how the costing of the vendors proposal will be 
accomplished. It is at this point that many of the previously mentioned elements 
Start coming together, e.g., the workload levels, extension factors, time requined 
to execute a benchmark, etc. The cost of money is introduced at this point. 

Miiere is a requirement to perform an economic analysis of the cost. 14 Currently 
fire ciscount rate in use is 10 per cent. 


An example of the formula used to define the cost of a system is shown 


below. The basis for this example is the workload levels and probabilities 


Mag S. , Department of the Navy, "Economic Analysis of Proposed Department 
of the Navy Investments," Comptroller of the Navy Instruction 7000. 28 ( April 19, 
1967) Wepariment of the Navy, Washington ©. Cc. 20290, 
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DESIRABLE FEATURE: Capability to expand al year 25 9he systems Sateen 
three activities to include time sharing systems for program testing 
purposes. 


ONE ES "Pime Sharing Versus Batch Processing: The Experimental 
Evidence" by H. Sockman, 10 October 1967, AD 661-665, Defense Documen- 
Gatton Center publication, reports that a 252. reduciionsie orcs an 
development time has been shown in experimental studies by using time- 
sharing techniques instead of batch processing techniques in testing. 
Time-sharing caused a 40% increase in computer test time. No signi- 
ficant differences were found in program size or run time. On this 
basis, the followings cost reduction would be gained from time-sharing: 


25% of call programmers’ yearly salary - 40% X hourly 
computey rental rate X hours test time per week K 32 
weeks per year = cost reduction/base. 


Taking number of programmers at each branch = 20 
Programmer (esi) eal = 220,000 

Hourly computer rental rate = $100 

Average Nours test time per week = 20 


Men (.25 X $20,000 x 20) —~ (.40 X Si00 KX 20 X 52) = cost reduciionee 
weal Lor one activity. i 


$100,000 - $41,600 = $58,400 yearly savings per activity 
eee re cc yitacs 
$175,200 total per year 
___—X4 years 
$700,800 total savings 


MOMPLET: Jf proposed, a value of 958,400 would be given for eaen activity 
per year for each year after the first year. 
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SOURCE: E.O . Joslin, “Competitive Computer Selection Within the 
Department of the Navy," unpublished paper presented to the Diebold Group Inc., 
October 1969. 
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Figure 5 ~~ Sample Value Statement 











shown on page 28. In this case, there is a 90 per cent chance of the workload 
being at Level 1 for the first twelve months, and there is a 5 per cent chance this 
will be the case in the second twelve months. This same type of analysis is 
performed for all of the cther workload levels. Based on this data the costing 


formula is then developed. The formula for the first worl) oad level would be: 


MCy n = monthly cost workload level n 
Y = number of months 
E = probability of operating at Pee be 
Dor = discount rate 
Mee,  F D+ YMC, ° P = D 
1Z2MC.,) x 0.90 x 0.91 + LMC.) : 0.05 = 10.33MC.) 4 


By performing the same mathematical operation to the other workload levels, the 
present worth of total expected lease payments for each configuration can be 
Expressed as: 


BeeooMIC.y 7 + 10.00MC.) 9 + 8.56MC.) 3 + 7.75MC. ges 6.89MC AF 


les 
f 5 
1.89MC.) Boe ke 37MC 7° 
The Selection Plan is now complete. The SSEB presents the RFP and the plan 


to the Source Selection Advisory Council for approval. When this review is completed 


and approval is granted, the RFP is forwarded to the suppliers. 


15y, 5., Department of Defense Computer histitute, Report of the Fourth 


Dp. 6. 
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Vendor Liaison and Validation 


ee 





The amount of effort that has to be devoted to this function is inverscly related 
to the degree of excellence in the RFP. The period of time is normally four months. 
Essentially all that is required of the Navy personnel is to be available to answer 
vendor questions, and to be ready to witness tlic running of the benchmark programs 
when the vendors are ready to demonstrate them. 

There is one cardinal rule in vendor liaison: whatever is told or applies to 
one vendor must be told to all vendors, and in writing. This rule is considered 
esscntial--it is often difficult to differentiate bctwccn liaison and negotiation. It 
is also necessary in order to avoid later difficulties which can arise when all 


bidders are not equatly informed. 


Proposal Evaluation 


ee ee 


Evaluation of the vendors proposals is not a simple task if, however, the RFP 
and Selection Plan is properly prepared the degree of difficulty will be reduced 
considerably. The actual steps included in the evaluation process are:16 


l. Reading and understanding all proposals. 

2. Verifying that all mandatory conditions are satisfied. If a proposal does not 
satisfy all manditory conditions it is ruled not responsive and it is no longer 
considcred. 


3. Determining the dcgree to which the requested desirable features have heen 
provided. 


—~ 


16yoslin, “Competitive Computer Selection,” p. 29. 
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4, Matching the response to the desirable features with the value statements. 
9. Costing the vendors proposal. 


The verification of vendor claims is often difficult. To assist in this, the RFP 
requcsts the vendors to reference the technical manuals. If the SSEB cannot verify 
a claim by using the technical manuals or other references, the capability in question 
is regarded as a vendor's promise and it is incorporated as a term and condition in 
any rcsulting contract. 17 

Determining the degree to which a desirable feature is satisfied and matching 
this with the value statement is also a difficult task. "It can result in many bruised 
egos, but normally it can be worked out to the satisfaction cf all." 18 

wie pricing or costing of the Mic end system is an involved process. The total 
cost is dependent on many elements: the number of years the systcm will be used, 
the number of hours the equipment will be used, the cost of the desirable features 
provided by the vendor, the cost of satisfying a desirable feature by an alternate 
means if the vendor does not provide the item, the cost of vendor support, etc. 

Another complicating factor is that each of these costs have to be considered 
under several different procurement possibilities: lease, lease with purchase option, 
purchase, and any special procurement plan which may be worked out with vendors.!? 
In addition and as previously mentioned, an economic analysis using the present value 


technique must also be included in the calculations. 20 





Wipia. Wipid, 


19bid., p. 31. 20NavComptiust 7000. 28. 
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Negotiation and Contract Awa rd 





The SSEB review of tle vendor proposals, is attended by a continuous search 
for items which might be negotiable. An example is an item offered by one vendor 
at no cost might be obtamable free from the other vendors if the item was subjected 
to negotiation. Usually such items as training, maintenance, and some software 
packages fall into this category. 

After the SSEB has defined which proposals are responsive to the requirements 
of thle RFP, letters arc nrenanca and sent to the vendors. The purpose of this letter 
is to set up the time and place for negotiation sessions with the responsive bidders. 
In addition the letter tells the vendor what items the Navy wishes to negotiate with 
the vendors. 

At the negotiation session, the vendors and Navy personnel discuss the items 
in question. The vendor is told when to submit the final proposals. These proposals, 
however, must be based on the original proposal plus information covercd in the 
negotiation sessions. Each of the contractors is encouraged to improve his proposal, 
and they are reminded to submit them by the scheduled date because late proposals 
might not be considered. The session is then closed, and a memorandum of the 
meeting is preparcd and becomes a part of the contract pre-award file. 2 | 


When the proposal changes are received, the SSEB again reviews the proposals 


2\oslin, (Ccumeutive Gouiputer-selection,.(p. <2- 
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and adjusts the cost data based on the changes reccived. The evaluation then 
determines the proposal representing the lowest overall cost to the Department 
of the Navy. The sclected proposal is nominated to the SSAC and to the Special 
Assistant to the Secretary of the Navy (SASN). 

When the SASN approves the selection, three copies of the contract are scnt to 
the vendor for signature. If the supplicr disagrees with some aspect of the contract, 
the point is discussed and resolved. The vendor's representative and the government 
contracting officer then sign the contract. 

Occasionally there is one other step. If the award is for more than $ 1, 000, 000, 
ADPESO does not announce the award for at least twenty working hours after notifying 
the Office of Legislative Affairs. ‘The purpose of this hold is to allow the appropriate 
Congressman to be told of the award. 22 

There arc several other actions that are initiated at this time. An announce- 
ment of the award is sent to the Commerce Business Daily for public announcement 
via that media. Lettcrs are sent to the unsuccessful bidders fidicnue that their 
proposal was not selccted however their future interest is solicited. This letter also 
provides the unsuccessful bidder an opportunity for a post- approval meeting with 
ADPESO. If any unsuccessful biddcr desires such a meeting a written requcst has to 


be sent to ADPESO within ten days. There are several guidelines established 


ae 


22thid., p. 33. 
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governing the conduct of these post-approval mcctings: 


1. No informal sessions with respect to approved equipment selection shall 
be held with any unsuccessful supplier, nor shall any oral information be 
furnished on the subject. 


2. All questions from an unsuccessful supplier shall be provided to ADPESO 
in advance and in writing, and should be as specific as possible. The DON 
meserves the right to reject irrelevant or immaterial questions» Converse: 
related and relevant questions which are obviously and naturally evoked by 
responses to previously submitted questions may be entertained within reason. 
In this regard, however, the normal provisions apply with respect to security 
Sensitive information and proprictary information. 

3. The judgement exercised in the application of sclection criteria to 
particular proposals is properly a matter wholly within DON purview. 


4. Post-approval mectings shall be held as soon after they are requested as 
practicable. 23 


23cocNavinst Ot ep. 1-9. 








CHAPTER IV 


ALPE RNATIVESSE PAG VONESYS@iivis 


To assist prospective users, today there are a number of different methods 
used in determining which computer system best satisfies the data processing 
yequirements of the organization. Some organizations sclect their computers by 
soliciting competitive bids while others select a system without seeking any vendor 
competition. When competitive bids are requested the criteria used tosderermaae 
the best system ranges from objective standards to subjective judgement. The 
purpose of this cliapter is to review the various current practices and explore the 


details of some of the systems used. 


survey of Computer Selection Practices 


Dr. Norman F. Schneidewind of the Advanced Systems Division, Systems 
Development Corporation, conducted a survey for Datamation and the results were 


subsequently published in the magazine.! The data was obtained by sending 


ti 


INorman F. sclineidewind, “Ihe Practice of Computer Seleerigm 
i (Pepruciry 19G/), pp..22- 25. 


Datamation, 
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questionnaires to four types of computer users. ‘The types of users and the number 


responses from each were: 


1. Commercial: Non-government, non-aerospace, non-university oi 
2. Government: Civilian, military, federal, statc, local 20 
3. Aerospace: Aircraft, missle, space 8 
4. University 4 
lotal responses 69~ 


The survey questionnaire was directed toward four aspects of the computer 


Eelection process. Ihe areas covered and the findings of the Survey are as iolig. = 


Competition 


The competition scction of the questionnaire was designed to determine whcther 
the selection process takes place on a competitive or sole source procurement basis. 
The responscs indicated that bctween 77 and 92 per cent of all users utilize 
competitive selection systems. The degree to which the various users rely on the 
competitive process is shown below. 2 

Government 94.4% 
Non-goverument 80.9% 
Commercial | 80. 6% 
Acrospace 89.7% 
University 73207; 
hom Aww sens 84.6% 


Dr. Schnceidwind determined that in general it was the user with a small installation 


(average monthly rental of $16,800) that used single source procurement, while the 


Ibid. 
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management of larger installations (average monthly rental of $62, 300) select their 


equipment based on a competitive bidding system. 


Methods 


‘The purpose of the methods section of the questionnaire was to determine which 
of the following selection methods are used. The methods are listed with the one 
considered by Dr. Schneidewind as the most objective first while the last was 
considered to be the lene objective: 

. Computer Simulation 

Mathematical Modcling 

. Program and Execute Test Problems 

. Evaluation of Benchmark Problems 

. Published Hardware and Software Evaluation Reports 
It was found that larger installations use more objective methods than small 
installations. The specific findings are given below. : 

0 $15, 000 $50, 000 


to to and 
$15, 000 $50, 000 Up 


Benchmark Problems Sve, Src, 76.9% 
Evaluation Reports 80.8 1 iO BS iats 
Test Problems 53.8 54.5 p12 
Simulation ay, Varo 46.2 
Modeling 0 4.0 30.8 


Tah 
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Selection miter 


The respondents were asked to rank the following computer selection criteria 
i@eorcder.of importance: 
Hardware performance 
Software performance 
Support provided by manufacturer 
Availability of application programs 
Compatibility with present hardware and software 
Delivery date 
Potential for growth (modularity) 
Cost 
The user was asked to rank these items by assigning the number one to the most 
important, the number two to the next most important,etc. According to 
Dr. Schneidewind the most objective criteria are hardware and software performance, 
while the least objective are availability ofapplication programs and manufacturer support. 
The results indicate that all users place more emphasis on the objective 
(hardware and software) criteria than on the subjective criteria. Dr. Schneidewind 
considered this to be the most significant finding of the survey. In addition the results 
indicate that commercial users rated cost as a less important criteria than did the 


: 4 
other users. The average rankings are shown below. 


Commercial Government Aerospace” Unineran 





Hardware Sie ail ales | Be yea} 
Software 2203 2.45 3.68 ers 
Manufacturer Support Bio wl 4,32 on Zo jy 50) 
Availability of Programs G57 F230 7.14 CARS 
Compatability 4,35 4.85 3.86 6.00 
Delivery Gn 6.45 6.43 fies 
Growth 4.68 4.25 Sync: 4.50 
Cost 4.38 4.25 2.43 Soto 


ee eee ee 


4 hid. 











Outside Assistance 


‘The intent of this section of the questionnaire was to dctermine the extent to 
which users depend on outside assistance in the seicction process and when they did 
what type of consultant was uscd. Spccifically the respondents were asked to indicate 
Which of the following types of outside assistance were used: 

Independent Consultant 
Accounting Firm 
Mancieerent Consultant 
Technical Firm 
Computer Manufacturer 

The findings indicate that the number organizations using outside assistance is 
small, and the organizations that do use consultants generally have small data 
processing installations. When outside assistance is used the computer manufacturers 
are relied on more heavily than the others. 

Based on the analysis of the data accumulated in this survey, Dr. Schneidewind 
developed what he considered to be a typical computer selection process. In general, 
computcr sclection is competitive, objective criteria is stressed, and the selection is 


performed in-house rather than by an outside consultant. 
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Review of Specific Alternative Systems 


The process of selecting a computer from among several alternative proposals 
logically breaks down into two major parts. The first part consists of the validation 
of the vendor's claims. ‘The validation process is concerned with verifying the 
capahilitics of each system proposed and the estimated system times required by 
each configuration to complete the workload. This is a very important part of the 
selection process. The user must be aware of the old saying "buyer beware." While 
the vendors will eibatel not perpetrate a falsehood the buyer must understand that 
the proposal is a selling device and certainly each vendor wants to look better than 
mimeriers. In order to look best, the vendors will interpret the specifications ime 
way that is favorable to their equipment. The amount of interpretation is related to 
iieeway the requirements are set forth m the specifications of the RFP. Ji aie 
specifications are loosely worded then the suppliers are put into a situation that 
requires a great deal interpretation of the specifications to make a proposal. An 
example of the types of problems that can arise if the proposals are not thoroughly 
validated against the organizations requirements is the situation at United Airlines 
and ‘Irans World Airlines: 

Moot the /aicesiscemimenrcinl COmpiten Sycicime ever Ordered =. tom 
reservations at United Airlines and Trans World Airlines—are not working out. 
United has decided to drop its Univac systein, estimated to be valued at more than 
?30 million, while TWA reportedly is close to canceling an equally large system 


ordered from Burroughs Corp. According to United, the Univac system was more 
than a year late and still inoperative. Industry sources report that delivery 
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problems in hardware and misunderstandings over programming tripped up 
Burroughs and TWA. Both airlines will continue to use their existing computers 
until they select suppliers for a third-generation system. 9° 
The second half of the sclection process is the evaluation of the proposals that 
satisfy all of the basic data processing requirements. The purpose of the evaluation 
is to differentiate among the various proposals and find the one that best satisfics all 
fmeeselection criteria. 
There are several methods, besides the ones used by the U.S. Navy, that can 
be used in the validation and evaluation of system proposals. The rest of this chapter 


ws 


purports to cxamine some of the alternative validation and evaluation systems. 


Alternative Validation Methods 
Two arcas of cach proposal must be validated: system capahilitics and system 
timing. 
Capabilitics 


Minewser cal Verily system capailities in several ways. First; aiiathe 
Memiitacturers technical literature can be read. The next step is to compare tlirs 


data with the vendors proposal. An alternate way would prepare a detailed 


2" Airline Computer Problems,” Business Week, February 7, 1970, p. 3. 
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questionnaire to be included in the RFP and request the suppliers to provide the 
requested data. ‘The vendors proposal is then compared with this data. A third way 
of obtaining this information is to study the Standard EDP Reports prepared by the 


Emerbach Corporation. 


Timing 


The validation of system timing is a most important task. In addition, it isa 
difficult task to secon sh, particularly with a high degree of accuracy. ‘The 
importance of this operation however, can not be overemphasized. ‘The timing 
estimates derived will serve at least two vital functions in the selection process. 
First this information will be used to determine if the proposed configuration can 
perform the stated wo1kload. Second, the timing data will provide the foundation for 
costing the proposal. 

There are several ways to verily the timing of computer systems: Denchinan sa 
computer simulation, and detail hand timing. The use of benchmarks was described 


fecnapter Itt. 
Simulation 
simulation is a technique in which a model of the actual system is described and 


constructed and testing of the real system is accomplished by exercising the model. 





OR dwa rd QO. Joslin, Computer Selection (Reading, Massachusetts: Addison- 
Wesley Publishing Company, Inc., 1968), p. 67. 
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The function then of a simulation model is to show the results created when certain 
‘ . = i - : Lf a. 4/8 7 a ° : ° me . 

things happen under the control of a set of decision rules.” The limitations and the 
advantages of simulation are stated by Teichroew and Lubin: 

Simulation problems are characterized by being mathematically intractable and 
having resisted solution by analytic methods. The problems usually involve many 
variables, many parameters, functions which are not well behaved mathematically 
and random variables. Thus, simulation is a technique of last resort. Yet, much 
effort is now devoted to “computer simulation" because it is a technique that gives 
answers in spite of its difficulties, costs and time required. § 

There are several computer- simulation techniques in use or under development; 

however, the system most widely known and used is one called SCERT (Systems and 


9 


Computers Evaluation Review Technique).“ SCERT has the capability to simulate 
over 100 different computer systems manufactured by the thirteen different companies 
companies. A complete list is shown in Appendix G. SCERT divides into four 
major components: definition laneuages, a factor library, simulation programs, 

uk. 
and output reports. 


The definition languages are used to define the applications systems to be 


simulated and the hardware/software complexes to be simulated. Several 


7Fohn EF. Cremeans, "The Trend in Simulation," Computers and Automation, 
January, 1968, p. 46. 





Deitel Teichroeu and John F. Lubin, "Computer Simulation-Decision of the 
Technique and Comparison of Languages," Communications of the ACM, IX (October, 
60), p. 724. 





Toslin, Computer Selection, p. 86. 


A Technical Description of SCERT (Rockville, Maryland: Comress, Inc.), 
pp. 17-49. 
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language section components facilitate this. The data definition division defines 

the files and reports of the applications system. The procedures division defines 
each application program. ‘The configuration division defines each equipment 
configuration to be simulated, including model number and quantity. The environment 
division defines the staff that support the installation plus salary level. 

The factor library contains the characteristics of the hardware and software 
items—characteristics such as cost, performance and technical specifications. This 
data is obtained by Comress from the manufacturers. 

The simulation programs perform the processing necessary to accept the input 
definition data and create the output reports.’ The programs are used in five phases 
which will be discussed later. 

The output reports consist of fifteen different reports. All reports are not 
created for cach evaluation; for instance, the real-time reports would not be 
produced when a batch processing system is being simulated. 


ie entire sGERT operation is showias Pioure 6. As can be seem thevern 


five functional phases. nt 


© Phase 1 - The Introduction of Processing Requirements 
The first input to SCERT is a series of definitions outlining the workloads and 
computer processing requirements of the system to be simulated. Specifically the 


imrOolination cntered is concerned witli three areis. 


—— eee 


Hl ; . . 
Unless otherwise footnoted, this and future information relating to SCERT was 
was generally obtained from A Technical Description of SCERT (Rockville, Maryland: 
Compress. Mic .)- 
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SOURCE: A Technical Description of SCERT (Rockville, Maryland: Comress, 
lies), ps lo. 


Figure 6 — The SCERT Program 
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systems Environment information relates to the environment in which the 
computer and data processing system will operate. The detail data elements are: 

Cost Data - Specifies the costs of operation such as programmer and 
operator salaries, and systems life expectancy. 

Programmer Experience Profile - the user indicates the experience and 
qualifications of the programming personnel. 

Definition Percentiles - the Wser indicates the percentage of the tor 
system requirements not defined for analysis by SCERT. ‘The simulator uses these 
Honp1oject the total. 

Keal Time - the parameters concerning real time processing team som 
emrered. 

File Definition defines each file in the data system. Each file is given a unique 
mune, the size of the file is specified by indicating the number of records; the 
number of characters and fields per record and the type of file is indicated. 

systems Definition is the common denominator for defining the processing 
requirements in the individual computer run or random event. Each run definition 
consists of three distinct levels: 

Run identification, irequency  priokity, and prerequisites. 2a cieiiniec 
defined by a unique number and the frequency is indicated as daily, weekly, monthly, 
quarterly, or yearly. 


File identification and through- put parameters. Each file is defined by the 
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unique number used in the file definition above. Any special considerations 
concerning the way the data is to be handled is indicated. 
The internal activity section provides the simulator with the internal 
processing applicable to the input and output files. 
Phase 1 has three primary functions: 
Accept the input definitions indicated above; 
Build a mathematical model of each computer run; and 
Validation of the model and output diagnostic data if any errors or 
inconsistencies are found. 
O Phase 2 - The Introduction of Hardware/Software To Be Simulated 
In Phase 2 the specific hardware and software configuration being appraised is 
introduced. This is a very specific and detailed definition of the hardware. The 
model numbers of every component, all special features, adaptors, number of 
channels, etc. to be used have to be provided. In addition, the details applicable to 
the software package have to be indicated. 
Pie tinetions ©! Pliase 2 a2 ¢, 
To accept the hardware/ software definitions; 
. To build a mathematical modcl representing hardware/software; and 
Yo validate the compatibility of the models built in Phase 1 with 
Phase 2 models. If discrepancies are found, a diagnostic report is 


created. 
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O Phase 3 - Presimulation Algorithms 
In this phase the models built in Phase 1 are passed against the models of 
Phase 2. During this processing a series of calculations are performed which in 
eifect structure and parameter the non-hardware oriented models to the performance 
abilities and capabilities of the hardware. 
During Phase 3 ihe following events for cach program occur 
The internal processing tinte and mcmory requirements are calcilated. 
. All files are assigned to I/O devices and channels. 
. All files are structured to meet the hardware parameters. 
. The thru-put time for all I/O and memory functions is determined. 
. the timing of the software packages is determined. 
The pre- and post-run times for factors such as setup-time, error- 
COrrection, elec. are calculmimucd. 
) Phase 4 - Simulation 
In this phase the actual simulation of the system takes place. All of the work 
performed in the first three phases was in preparation of the data for input to Phase 
ieee Liere are three distinct stages: 
Stage 1 - Each program run or random event is expanded into its maximum 
number of unique thru-put iterations. The processing simulates the flow of each of 
these thru-put iterations through the computer configurations. The results are the 


net thru-put timing. 
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stage 2 - This stage is entered only if the defined processine imehides 
random-occuring events. This stage constructs models based on probability theory 
and distributes the occurrences of events in a probabilistic manner. These models 
are then simulated and potential queueing points are determined, and processing 
werays caused by the quches are derived. 

Stage 3 - This stage is entered whenever the hardware/software complex 
is capable of multi-programming or multi-processing. In the first two stages the net 
elapsed time and capacity requirements were developed. It is the function of Stage 3 
to schedule the concurrent processing of these events. When this is completed, the 
results stored in the mathematical models is ready to be printed out. 

O Phase 5 - Production of Output Reports 

SCERT can provide the user with filteen standard reports. If requested the 
simulator can be modified to provide nonstandard or special purpose reports. The 
reports that are of particular interest in validation are given below. A complete list 
of the standard SCERT reports is shown in Appendix H. 


Computer Complement Report—Portrays the exact configuration that was simulated 
and provides certain basic cost data about the configuration. 


Central Processor Utilization—Summarizes for each scheduled run the projected 
running time and horizontal memory utilization. 


Cost Summary—Primary cost output which relates the projected utilization of the 
computer to lease, purchase, and maintenance cost considerations. 


Detail Analysis—Produced for each scheduled run or real-time event simulated 
and serves as backup for all other reports. It precisely portrays the utilization, 
timing, and memory requirements derived by the simulation and presimulation 
algorithms for all components making up the run. 
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Hand ‘Timing 


‘The essence of hand timing is to analyZe cach and every function that the 
computer system will be required to perform during the running of a program. The 
hardware cnginecring time for each operation is determined and all of these times 
eresraced 1Occtitento derive the program running time: 

This method of timing requires the user to have a detail knowledge and 
understanding of the programs that are going to be timed. Jn addition, a 
standardized method of eulheseine the required data should be used. This is necessary 
in order to obtain consistency between vendors and limit the number of approximations 
in the timing data. Some suggested timing tables are proposed by Joslin. 12 There 
are several data elements that should be in a detail time table: 

Program Number and Frequency 
. Input/Output Requirements 
- Central Frocessor Memory Requimenrents 
. Non-productive Time 
Productive Time for Each Component 
In addition to a detail sheet the user should have a summary sheet. On this form the 
user can enter the data from the detail sheets. From this data the user can then 


determine the monthly run time. 


12 yoslin, Computer Selection, p. 83. 
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Alternative Evaluation Methods 


The purpose of the evaluation process is to define which of the several proposals 
submitted is best suited to the organizations needs. Determining which is best 
forces the user to establish some criteria that can be used as a yardstick to 
measure the proposals. The objective of evaluation is usually to sclect a 
proposed ADP system, capable of accomplishing the future workload at the lowest 
total cost. 

iere ake several different ways for evaluating the proposals. The alternatives 
range from a policy of notrocking the boat and staying with the current supplier to the 
very competitive situation that uses objective criterion in the selection process. In 
order to serve the purposes of this paper only the systems considered to he objective 
will be reviewed. 

The literature search for this paper revealed three different objective methods 


for the evaluation of vendor proposals: Weighted Scoring, Cost-Effectiveness, and the 


Cost- Value techniques. Cost-Value was reviewed in Chapter III. 


Weighted Scoring 


; : : ] 
There arewseveral different welehted scoring systems that cam be user: 3 


ee 


1 Shox the interested reader: Eugene S. Schwartz, "Computer Evaluation and 
Selection," Journal of Data Management, VI, No. 6 (1968), pp. 58-62; and, J.A. 
Campise, "A Quantitative Approach to Equipment Comparison," Journal of Data 
iMianocement,. 1 No. o (1963), pp. 12-20, 
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For this paper, the system proposed by Mr. Solomon Rosenthal will be reviewed. 14 

The first step, when using a weighted scoring system, is the establishment 
of the selection criteria. ‘The basis for developing this plan is the systems 
specification of the information system. ‘The user must define those characteristics 
of an ADP system that will be required to satisfy the needs of the organization. The 
specific items included and the relative weight assignment to cach item varies based 
on the requirements of the information system. In Mr. Rosenthal's system, the 
factors are classified as major, intermediate, or minor. 

The major categories include factors such as: overall costs, equipinent 
characteristics, systems potential, vendors support and others as required. ‘The 
intermediate and minor categories would include the component clements that make 
up the major area. For example, within the cost element, details would focus on 
the cost of individual components and features of each. In addition, costs relating 
to the installation of the equipment and its subsequent operation would also be 
considered iu the intermediate or minor categories. 

The next step is to assign weights to the factors to be used to evaluate the 
proposals. It is important to note that if this system is to be impartial, the weights 
must be assigned before any vendor proposals are received. In fact, the weights 


should be assigned before the RFP is circulated. The maximum score possible is 


4c olomon Rosenthal, “Analytical Technique for Automatic Data Processing 
Equipment Acquisition,” Proceedings of the AFIPS Spring Joint Computer Conference, 
re a 
1964, pp. 359- 366. 
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established as 100. This score can be achieved only if one proposal is the best in all 
factors. A part of the 100 maximum is assigned to each major factor based on the 
relative importanee of each. Then weights are assigned to the intermediate and 
Minor factors. The sum of the weights assigned to the intermediate and minor 
elements must be cqual to the appropriate major factor of which they are a part. 


The results of this process might look as presentcd in the following example. 





FACTORS WEIGHTS 
Cea 35 
Machine Rental 20 
One-Time Costs 10 
Personnel Support Cost fe) 
Equipment Capability 295 
Random Access Storage 10 
Systems Validity Checks = 
Processing Speed 10 
Systein Potential 15 
Ability to Increase 10 


Random Storage Rapidly 
Ability to Augment 


Central Processor 5 

Manufacturer Support 10 
24-Hour On-Site Maintenance 95 
Back-up in Vicinity 5 

Common Language Programming 10 
COBOL 5 
FORTRAN IV 3 

Physical Requirements 3 


After the proposals are received and validated, they are ready for scoring. It 
should be noted that when using this system the individuals involved in the validation 
of the proposals do not do any scoring. The scoring is accomplished by a completely 


different group of people. In addition, the scorers do not know which vendors’ 
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proposal they are processing. In fact, the scorer does not even see the proposals. 
The scores are recorded on an evaluation sheet that was prepared in the validation 
process. The evaluation sheet shows the data applicable to each factor. In addition, 
the validators prepare a written report indicating the costs, processing times, and 
differences between the proposals and the specifications. Is 

Based on this information, the scoring begins with the minor elements first, 
then intermediate scores are developed, followed by the majors and finally the grand 
total. The minor scores‘are developed by either of two methods. Some minor factors 
are not compared one proposal with another, but are assigned a predetermined score 
if certain conditions exist. For example, if free maintenance is provided twenty-four 
hours per day, “X" points might be awarded, if only for cight hours per day, a lessor 
score would be assigned. 

The second method would apply to the majority of the minor elements. If the 
smallest is best, the score is computed as follows: 


(comparison base) (maximum possible) 


= minor score 
(this vendor entry) 


If the largest is best, the following formula is used: 


16 


(this vendor entry) (maximum possible) 


minor score 
(comparison hase) 


IS thid, MOnia. 
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The next step is to total the minor scores within each intermediate. ‘The 
intermediate scores are added together to form the major scores and the final score 
ms ile Sumi of all the major scores. 

The proposal then with the highest score would be recommended to top 


management as being the best system to select. 


Cost Effectiveness 


The concept of cost effectivenessisprimarily attributable to the work of Hitch and 
McKean, 17 and the principlesofthis concept provide the foundation for a computer 


selection system developed by the Mitre Corporation. Prior to analyzing this system 


however, it will be helpful to define the words cost and effectiveness. 18 


Effectiveness - The degree to which a system will perform the future jobs and 
satisfy the constraints. Effectiveness is generally considered to consist of the 
following three main components. . . | 

a. Capability - The degree to which a system will perform the future jobs and 
satisfy the constraints, assuming that the system is always available for operation 
and will never malfunction. Capability can be measured in various ways but the 
two key measures of capability are: 

(1) Quality of the work output. This measure is in general multi- 
dimensional since it encompasses the many sub measures used to measure 
the work output. For example, it might include the straightness of a line of 


pili or (Ne Masi Wumber ol Copies ol printout: 


Nuclear Age (Cambridge, Mass.: Harvard University Press, 1963). 


187 D. Porter and B.H. Rudwick, Application of Cost-Effectiveness Analysis 


Gar 8 ama eee 


to EDP System Sclection (Springfield, Virginia Clearingliouse: U.S. Department of 
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a 


Comme. ce, ADV60/-522, October, 1967). 
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(2) ‘Time. Given that the quality of the work output can be measured, a 
second key measure of the EDP system capahility is the time taken to 
perform the future work load, again assuming the system is available for 
and never malfunctions. For example, the measure could be the expected 
time to perform a given monthly workload. 
b. Availability - may be defined as the probability that the system will be ready 
for operation when called upon. 
c. Defindability - may be defined as the probability of the system completing 
the job satisfactorily, given that it was available. 
Cost - The total dollars required to procure, operate, and maintain the system 
to perform the future sct of EDP jobs. . . .19 


J ° 


Since these two elements have been defined it is possible to measure each one 
Peparatcly. The Dope then becomes one of combining these two elements ina 
way to facilitate decision making. To do this it is necessary to specify in advance the 
level of either the cost or the level of the effectiveness. The mechanics are explained 
below and illustrated in Figure 7. 

Level of effectiveness specified: called pivoting on constant effectiveness. If 
level Ey is selected as the comparison level, system A could not be selected. If the 
effectiveness of A were increased to Ey its cost would then be at cither point Aj or 
Ag. Now system A and B can be compared with each other. If the A system cost 
were Ay then it would he selected; however, if the cost were at A3 then the B system 
would be selected. 

Level of cost_specified: called pivoting on constant cost. If the Cy cost level is 
selected as the comparison level of cost, and the cost of system A is increased, 
its level of effectiveness moves to either Aj or Ay. When A's effectiveness and cost 


own ee ee ee eee ee 
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Cost 


SOURCE: J.D. Porter and B.H. Rudwick, Application of Cost-Effectiveness 
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Analysis to EDP system selection, U.S. Deparinient of Commerce; AD 66 725722 








Figure 7 —- Measurement of Cost and Effectiveness 
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intersect at Ay the B system is the better buy. The converse is true when the 
effectiveness and cost of the A system is represented by the point Ag. In this 
situation the A system would be the better buy. 

The first sclection criterion, pivoting on constant effectiveness, is more 
applicable to the problem of computer sclection, so the cost-effectiveness system 
devised by Porter and Rudwick is based on this principle. 20 

There is a great deal of similarity betwcen the cost-effectiveness system and 
tie cost-valuc system used by the Navy. Both methods are based on: a probabilistic 
workload projection, mandatory requirements, and desirable features. 

There are differences, however in the way the projected workload is derived and 
in the way that value is calculated for the desirable features. Both systems however 
regard the mandatory items in the same way. The mandatory features are validated 
and the requirements in the RFP concerning these items must be satisfied for the 
Paeoposal tobe considered responsive: 

The total selection process using the cost~cffectiveness system is shown in 
Appendix I. 

Probabilistic workload projection: the basis for using probabilities in 
projecting future workload is the fact that the user can predict with a high degree 
ecoumcdcnce certain “kitewn jobs ;lewever,thereciarc "likely jols simielivean 


only be expressed quantitatively with a lower deeree of confidence. zl 


Uibid., p. 17. 2\nnid., p. 18. 
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The user preparcs a projection of his future workload by starting out with a 
diagram similar to the one in Figure 8. The lowest line is called the "Reference 
Workload” and the user has indicated that there is a 100 per cent probability that this 
workload level will be experienced or exceeded. The user then assigns a probability 
to each of the other workload level projection lines. 

The next step is to average the yearly workload levels. ‘This, then, permits a 
diagram like that shown in Figure 9 to be drawn. The purpose of this is to simplify 
mie cost calculations. 22. 

In the validation phase, timing data was developed for each system proposed. 
This information is correlated with the various workload levels. If benchmarks were 
used this correlation is accomplished by using the extension factors. 

The evaluation team can now construct a chart similar to the one shown in 
Figure 10. In this situation the RFP had a mandatory requirement that the workload 
be completed in 600 hours. In view of this, Vendor A has proposed two systems: Aj 
and Ag. System A, , a cheaper system, can satisfy the mandatory requirement until 
somctime in the third year of the system life. Then it is augmented to become system 
Ag, which can satisfy the 600 hour requirement for the aouetanler of the systenmiice 

It will be noticed in Figure 10 that the method of expressing the probability is 


different. The figures now represent the probability of the workload falling within the 


mespective secameiits. 


22 
PORTCmnG IMUenmicn, = COSst Eifectiveness, p. 23. 
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Ficure 8 — Probabilistic Workload Projection Smoothed Over Time 
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Figure 9 — Probabilistic Workload Projection Averaged Per Year 
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Figure 10 — Projected Time for System A, and A, Proposed By Vendor A 
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The cost of each system can now be calculated. This is done by determining 


tie yeariy COSt Of Operating Gach sy siemruisiollovs: 


Cd oe Coal 

P, = Probability of workload in 
Seament | 

C, = Cost of operation in segment 1 

c z PyCytPoCot. ¢ ee 


After the yearly cost is determined the total discounted present value is derived in a 
manner similar to the one shown in Appendix J. 23 

Desirable features make up the third category of the user's data processing 
requirement. Porter and Rudwick are concerned with the necessity of this third way 


of expressing data process requirements because: 


i, The representative workload only approximates the actual workload. 
Since there may be other workload requirements that will not be measured in the 
systems timing determination, by including desirable features the uscr is provided 
a liedce against this uncertainty. 

I The inclusion also provide a hedge against the unccrtainty attendant to 


* ete iz 
measuring a systems ability to perform the future workload. at 


23 .S$., Executive Office of the President, Bureau of the Budget, Discount Rates 
and Procedures to be Used in Fvaluating Deferred Costs and Benefits, Circular A-94, 


rn Pap ea B,D, 


Washineton, | De a. C., Attachment A. 








SO RA AR NS a Oe et enema kee ee 


24 , 
POmlet anc Kuchvick, Cost biteclivencss, p. 18. 
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os The use of desirable features permits a reduction in the number of 
mandatory requirements. By definition the vendor must satisfy all mandatory items 
or his proposai is eliminated. This could create an undesirable situation. For 
example, suppose the RFP contained a mandatory requirement for the proposed 
system to have a two second response time. Vendor A's system has a response 
time of 2.1 seconds, but that system costs 20 per cent less than all others. It is 
difficult to defend the need for that additional one-tenth of a second advantage, 
particularly in view of system A's cost. 

4, The manufacturers of data processing equipment are competing with each 
other. This spawns technological innovations which improve the design of computers 
and the attendant software. The use of desirable features in the RFP provides the 
vendor an excellent avenue for proposing these advances to the user. Without these 
features the user would be denied the opportunity of evaluating these advanced 
capabilities. 

Once the desirable features have been stated it is then necessary lo provide a 
way to evaluate the vendor proposals. There has to be a way to relate the character- 
istics of any particular desirable feature to the job or jobs that will be improved by 
the use of this desirable. The question then simply stated is: Is the desirable feature 
oifered by the vendor worth the price he charges? ‘This question cannot be answered 
unless a value of this worth has been assigned. In order to assign a value, however, 


Hie term Word: has to be déiined. in this system, the worth of 2 teamine st. equal to 
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"the lowest incremental cost to do the same job if the feature is not available. "4° 

There are two ways provided for determining and evaluating the worth of any 
desirable feature. The first way dctermincs worth through an analytical process 
and the second method makes use of comparative ranking. 

Analytical determination of worth: Since the worth of a feature is defined as 
the lowest incremental cost of doing the job if the desired feature is not available 
from the suppliers, it is necessary to first identify the alternative ways of doing the 
job. Figure 11 depicts the altcrnatives that are available to the user. The next 
step is to determine the cost over the life of the system of alternative ways of 
accomplishing the function. The worth of the feat ure is the least of these costs. 

It is also necessary to identify the cost of the alternative ways of obtaining the 
feature and doing the job using the feature. For example the feature might be procur 
procured from othcr sources or, in the case of software the user might develop the 
capability using in-house resources. 

Based on this compilation of data, it is then possible to determine the way to 
accomplish the function at the lowest total cost. This is referred to as the “efficicnt™ 
Ppivition.°° Restated, the efficient solution is found by determining the lowest cost 


method of obtaining the feature and doing the job, then comparing this cost with the 


worth, and choosing the lowest cost alternative. 


29ibid., p. 33. 2Otnid., p. 34. 
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Determination of worth by comparative ranking: There may be times when it 
will not be possible to determine the worth of all desirable features by analysis and 
considcred judgement. In these instances, intuitive judgement is substituted as a 
part of the quantitative evaluation. The steps to implement this procedure are: 


A 


(a) All desirable features are ranked in order of importance. 

(b) Points are allocated to each feature thus establishing the relative worth of 
each. 

(c) The points are translated into dollars. This is done by calibrating one or 
more of the features by determining its worth on the analytical basis descrihed 
aDOve. 

(d) The results derived are reviewed using intuitive judgement as the sounding 
board. If discrepancies are found then either all or some of the items should go 
through the whole process again. 2/ 

Yt is now time to bring together the information derived in the various areas, 

so that a composite picture of cach proposal-can be developed. To do this a 
worksheet like the one shown in Figure 12 will be used. In this example there 

mie three vendors (A, B, and C) proposals beime evaluated. There are three 
geeiiable fcatures to be considercd—Fj, F959, and ee The users worth and the 
least cost for each is shown in item 3 on the Evaluator Worksheet (Figure 12). 

This illustration demonstrates how the costs are derived if the vendor does not 
provide the feature at all, or the feature is provided in the basic system and cannot 
be excluded, and finally the instance where the vendor does propose the feature and 


states its cost. 


The initial entry on the worksheci is the Total Proposed Vendor Cost. This 


27}bid., p. 37. 
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Figure 12 —~ Evaluator's Workshect 
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represcnts the cost of each manufacturers proposed system, if the system is 
accepted as proposed. Jt also is the cost as calculated by the vendor. 

The evaluator enters the previously derived cost data to process the workload 
of the life of the system, and thus derives the expected cost to do a representative 
workload. 

At this time the desirable features which each vendor has proposed and the 
incremental costs associated with each are entered on the worksheet to determine 
the cost of additional job benefits. In this example note that vendor A does not 
fmovide any of the threesfeatures. In the case of vendor B, feature F1 is melidédan 
the system cost while F9 is not provided but I9 is at the cost indicated. Vendor C 
has provided all of the features. The cost of feature f'g however, is not quoted 
separately but included in the system cost. Now the evaluator can determine for 
each vendor the least costly of the three alternative ways of receiving the benefits 
provided by each of the desirable features. ‘The three alternatives are: 

a. Buying the desirable feature from the vendor (at the vendor's proposed cost). 

b. Obtaining the desirable feature from another source (at the least cost of 

feature if obtained separately). 

c. Not buying the feature, but using the least costly alternative way to provide 

the benefits (at a cost equal to user worth). 28 
The evaluator now enters the values representing Users Worth and the Least 


Cost. These values are calculated prior to the actual evaluation process. 


The lowest additional user cost for obtaining the desirable feature can be 


28Porter and Rudwick, Cost Effectiveness, Dama oa. 
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determined. Note in Figure 1Z the cost data entered for the three desirable feamme-= 
In the case of Fy, the User Worth is 510K, that is the user can perform the jobs 
associated with this feature at an expected cost of $10K. Since vendor A does not 
provide this feature, the user will be forced to spend $10K in addition to vendor A 
costs to satisfy the requirements associated with F,. Vendor B includes this feature 
as part of his basic system and the cost of it cannot be segregated. The user then 
will not have to spend the 10K if vendor B's system is used. Vendor C can provide 
}*) however, the cost is $15K. Thus, the evaluator would eliminate the feature from 
vendor C's proposal since its cost is higher than its worth to the user. This same 
type of analysis is applicd to all of the other desirable features. 

The total expected cost to the user is then calculated by adding the cost of 
mre cCsirable feature, or the user cost equivalent, to the expected cost todo the 
representative workload. This cost completes the cost calculation. 

There are some points worth noting concerning this method of arriving at thic 
final cost figures for the proposals. Vendor A had the lowest proposed cost and the 
lowest cost to perform the representative workload. On the other hand, winning 
vendor C had the highest proposed cost and the highest cost to perform the workload. 
These costs however, do not cover all of the users requirements. ‘The total cost is 
the only one that will provide for satisfying the total data processing requirement of 
the user. Accordingly the total cost fioures are the true criterion for making the 
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EVALUATION OF THE NAVY SYSTEM 


It is the purpose of this chapter to evaluate the Navy's compective computer 
selection system. The review will be conducted by first appraising the organization 
responsible for selection and then by looking at the various stages in the process. 


The Automatic Data Processing Equipment Selection Office 


The Secretary of the Navy en aniened ADP ESO on lei 196) liaise esc cial 
to appreciate the role played by this organization if one is going to evaluate the 
Navy's computer selection system. This becomes abundantly clear when one 
considers the fact that prior to the establishment of ADPESO there was no one 
system in use. The selection of ADPE prior to ADPESO was accomplished by 
the various heads of departmental components. The procedures used varied 
depending on the philosophy of each of the departmental compenents. There was 
videspread use of ad hoc committees that were formed as the need arose. The 
contracting was conducted by a regular contracting officer, not one specialized in 


ie procurement of ADPE. 
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The establishment of ADPESO thus acconzplished two very significant things. 
First the responsibility for selection of ADPE was centralized in the Navy. Secondly 
a full time staff was recruited to carry out the mission of ADPESO. 

In addition to centralizing the fuiiction, the function was also elevated to a 
mower level in thle Department of the Navy. The Director of ADPESO reports to the 
Special Assistant to the Secretary of the Navy. Thus, top management attention is 
always focused on this vital function. 

There are twenty-two professional members on the staff of ADPESO. ‘These 
individuals are responsible for developing the procedures used, plus performing the 
functions of equipment selection. There are several significant bencfits derived from 
this. First, and probably most significant, is the fact that these individuals are 
professionals. This means that the breadth and depth of knowledge of each individual 
presently involved in the sclection process is many times greater now thau was the 
case prior to July 1967. This increased knowledge has a very profound impact on the 
RFP's sent to vendors and the appraisal of the proposals submitted by the supplicrs. 

In the years prior to the establishment of ADPESO, the specifications and RFP's 
were prepared by the user activities. The degree to which these spccifications were 
not oriented toward any given manufacturer was dependent on the experience of the 
individuals preparing them. Since these individuals were normally not involved fuli 
time in the selection process, their range of knowledge was limited. In vicw of this 


it was possible for the specifications to be oriented one way or another. Thie full time 
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professional staff at ADPESO does not labor under this handicap so the package sent 
to the vendors is not susecptible to this flaw. 

When the vendor's proposals are received, the full-time professional staff is 
responsible for reviewing them. From time to time, user personnel do assist in the 
appraisal process, but this cffort is still under the close control and direction of 
ADPESO. As indicated in Chapter IW, the group actually responsible for preparation 
of the RFP and the review of all the corresponding proposals is the Source Selection 
Evaluation Board (SSEB). It is important to note that the SSEB is constituted for 
the entire life of equipment acquisition. This means the individuals can become 
intimately acquainted with the requirements of the individual cases. This 
comprehensive understanding of the user requirement coupled with specialized 
knowledge of the individuals on the SSEB ereate a climate favorable to rational and 


objective evaluations. 
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The contracting officer is a key ADPESO employec. He devotcs all his efforts 
fo contracting for ADPE. By specializing in the proewurement of ADPE, this jimdinvidial 
has developed an in-depth knowledge, understanding, and appreciation for the 
particular problems associated with acquisition of the equipment. The results are 


substantial savings for the Navy as shown in the cases presented in Figure 12. 


Evaluation of Selection Procedures 


In Chapter III the Navy's selection process was divided into five subcategorics. 
Accordingly, the evaluation of the system will be done by examining cach of these 


erCas. 


oystems Specifications 


First, it is significant to point out that prior to the establishment of ADPESO, 
there was no standard method by which a user could document his data processing 
Fequirement. Each level in the user's chain- of- command had its own unique sct of 
requirements concerning what information should be included and the format in which 
the specifications were to be presented. Accordingly ADPESO's most important 
initial contribution to computer selection in the Navy was to promulgate instructions 


that standardized the format of the specifications. 1 


ese 


In. oO. Joslin, “Competitive Computer Selection Within the Department of the 
Navy," (Unpublished paper presented to the Dicbold Group, Inc., October, 1969), 
De o- 
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The specification review conducted by ADPESO is very extensive. This review 
plus the standardized format provides the Navy with an objective sect of requirements 
for submission to the cquipment suppliers. 

It is significant to note that it is the Navy's policy to minimize the numbcr of 
mandatory requirements contained in the specifications. 2 Every item that is 
classified as mandatory is carefully reviewed. When the justification does not 
substantiate the claim the item is reclassified as a desirable feature. This is 
done so that the specifications will not become so stringent as to either discourage 
vendor participation or to cause an undue number of proposals to be classified as 


mom responsive. 


The Selection Plan 


The selection plan consists of the detailed instructions that will be used to 
evaluate the proposals submitted by the vendors. The significant feature of the 
Navy's philosophy here is that the plan must be prepared before the REP is issued. 3 
This means that greater objectivity is injected into the evaluation process. If the 
plan were not prepared until after the initial review of the proposals, there would 


be danger of personal preference creeping into the plan. 


2U. S. , Department of the Navy, "Specifications, Selection and Acquisition of 
Automatic Data Processing Equipment, Secretary of the Navy Instruction 10462. 13, 
Department of the Navy, Washington, D.C. 20390, p. B-11. 


SIbid., p. III-5. 





Vendor Liaison 


The ADPESO has defined some very meaningful rules that govern the 
relationship between its representatives and the various manufacturers. ‘The intent 
of these rules is to establish an impartial and objective way to deal with the vendors. 
These rules have been provided to the vendors so that they know how to proceed in 
their dealings with the Navy personnel. One very noteworthy policy is that 
information provided to one supplicr is provided to all and in writing. The 
comprehensiveness of ue policies concerning supplier relations plus the fact 
that ADPESO is a continuing body stafied with professionals creates a climate for 
fair and objective interface between the vendors and the Navy. 


Validation of Vendor Proposals 


The validation of the vendor's proposals is concerned with two areas. First is 
the verification of the proposed systems capabilities and the second is validation of 
the system timing. The Navy uses several sources of data to verify the systems 
capabilities. Pune, the vendors are requested to provide technical data in their 
proposal. In addition ADPESO has manufacturer technical manuals and the Auerbach 
srandard EDP Reports. There is however, a dactor that seems to be even more 
important than these aids--the professional staff at ADPESO. Because these people 


are continually dealing with the subject matter, their ability to appraise the vendor 





85 


claims is certainly greater than what could have becn expected prior to the 
establishment of ADPESO. 

The primary method of validating the system tiining is through benchmark 
programs. In Chapter JV two other methods of system timing were discussed— 
computer simulation and hand timing. There are advantages and disadvantages to 
each of these methods. In certain instances it is possible that any one would be the 
best considering all factors. Joslin belicves however, that the prime consideration in 
choosing a timing technique should be the degree of confidence required in the data 
derived. The following chart indicates the levcls of coufidence that can be obtained 


with the three methods. Notice that the confidence levels are shown covering a wide 
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range. This is necessary to allow for variations in the applications and personel 
using the tecliniques. The information indicates that the highest levels of confidence 
can be placed in the results obtained from the use of a benchmark program. 
There is another significant advantage of using benchmark programs-—the 
programs are run on the system proposed by the vendor. ‘This gives the user an 
opportunity to determine the effect on the application of such features as: Ifo control, 
central processor administrative and processing functions, cffective specd of the 
various I/o devices, system simultaneity, the software scheduler, and the efliciency 
of the programming system in using the hardware capabilitics. . 
Another very important item to consider is the cost clement. The U.S. Army 
in 1968, conducted a study designed to determine the advisability of continuing an 
effort to program a computer simulator. In the course of this review, cost data was 


developed concerning three timing techniques. The Army found the cost per project 


for obtaining timing data was: 


Hand Timing $ 34, 124, 
Simulation ty Cin lov dn abate 
Bencliunmarks $29,548. 6 


450. Joslin, Computer Selection (Reading, Massachusctts: Addison-Wesley 


Publishing Company, 1968), p. ioe 





“Ibid. , p. 78. 


Ou. S., Department of the Army, Computer System Support and Evaluation 
Command, Report of the $? study Group System and Software Simulator (Washington, 


ec.: December 20, 1968). 
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Based on this plus. otlicr considerations in the study sille Army decided noite 
continue devcloping a simulator. Instead, benchmark programs arc bcing used. 
There are disadvantages to using the benchmark programs however. 
Before benchmark programs can be used the actual programs have to be written. 
This task takes time. The amount of time varics with the complexity of the 
programs, the knowledge and expericnce of the programmers, particularly the level 
of experience with programming languages such as COBOL, FORTRAN, or JOVIAL. 
This is so becausc these languages generally can be processed by all the different 
computers. This time clement could cause the user a loss of a state-of-art condition. 
The influence of this factor should decrease within the Navy in the future for the 
trend is toward using these highcr level languages as a mattcr of course. In light 
of the advantages stated perviously, plus the fact that the DOD now requircs all 
services to use benchmarks, it appears that the Navy's usc of this technique makes 


mare best ofa difficult task. 


Evaluation of Vendor Proposals 


The method used by the Navy to evaluate vendor proposals is based on the Cost- 
Valuc Technique. 7 In Chapter IV, two other evaluation systems were reviewed— 


Cost-Effectiveness and the Weighted Scoring methods. There is a great deal of 


oslin, Comlniiem pc echo pa alc: 
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similarity betwecn the Cost-Value and Cost-Effectiveness systems. One difference 
is the way the worth of a desirable feature is derived. Another difference, only slight 
however, is in the way the workload projection is handled. 

The Cost-Effectiveness system does not provide for the probability of the 
workload falling below the projected reference workload line. In the Navy's system, 
the user indicates a projected rate of growth and then assigns a probability to the 
levels above and below the trend line. This approach seems to be more realistic. 

The significant advantage of the Cost-Value system versus the Weighted 
Scoring method, particularly to top management, is that the results are expressed in 
dollars instead of points. By expressing the evaluation of each vendor proposal in 
dollars, the senior managers can readily understand what is happening in the 
evaluation process, but; there is no real need for these people to understand complex 
and technical data concerning each proposal. 

Another important characteristic of the Cost- Value technique not found in the 
Weighted Scoring system is the ability of the user to analyze cost associated with 
desirable features. The user can look at worth and cost of a desirable feature and 
make a sound procurement decision. This capability is particularly important in light 
Oi tie new pricing policies of ADPE manufacturers. 

While the Weighted Scoring system appears to be objective, there is a very 
significant crack in its wall of objectivity. The factors chosen as the evaluation 


criteria and the assignment of weights to them is subjective not objective. There are 
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no objective guidelines to assist buyers in matching requiremcnts with factors or the 
relative importance of each. It is unlikely that any two analysts, given the job of 
independently establishing appropriate factors and weights to select the best computer 
for a particular installation, would arrive at similar conclusions. There also isa 
possibility that the weights could be juggled and thus lead to almost any result. 
Ramtaer, there is a real danger that by the time the analyst has pesiemmed all the 
necessary weighting and scoring, he might lose sight of their shaky subjective 
foundation and attach. undue significance to the results. ° 

In view of the above factors, it is believed that the Cost- Value technique is the 


best method of evaluating the vendor's proposals. 


Sugee sted Improvement 


While the Navy's selection system is excellent, there is one area that could be 
made even more meaningful. Currently the method of costing the vendor proposal 
results in a figure that is the most probable. This figure is derived by the metnod 
described in Chapter III. Basically the calculations involve a monthly workload cost 
at various levels plus the probability of operating at these levels and a discount factor. 

Mr. David Hertz proposes a way of informing the executive of the uncertainity 
surrounding capital investment decisions. ? In this system, probabilities are assigned 


ee 


Stohn R. Hillegass, "Systematic Technique For Computer Evaluation and 
Selection," Managemeut Services, VI (July-August, 1969), p. 39. 


David B. Hertz, "Risk Analysis in Capital Investment,” Harvard business 
Review, XXXXIL No. 1 January/February 196+), p. 95. 
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the various possible outcomes. The cash flows connected with cach of these 
possibilitics is calculated. These valucs are then uscd to determine the expected 

net present valuc. ‘This data is then portrayed graphically. The significant 
advantage of this is that it allows management to sec the expected cash flows plus 

the variability of the flows and the risk associated with each. The present system 
contains all of the data required to perform such an analysis. Jt would be neccessary 
however, to obtain a computer program to perform the required calculations. If this 
type of analysis were incorporated inio the Navy system it is believed that there would 
be substantial beucfits derived. Accordingly it is recommended the Navy conduct a 
study and investigate the feasibility of using a-risk analysis technique in its evaluation 
wirOcess. 

Since ADPESO was estabtished, cxpericnce with centralized competitive 
Selection of computer systems in the Navy has demonstratcd that: the using activities 
are happier, the vendors arc happier because they fcel they are being given a true 
chance to bid, and the cost of selection within the Department of the Navy has been 
reduced 50 per cent while the numnber installations has increased 30 per cent. Thus, 
centralized, competitive computer sclection in the Navy has proven to be very 


satisfactory. 10 


10, 
De 3). 


oslin, "Compctitive Computer Selection Within the Department of the Navy,” 





CHAPTER VI 


CONCLUSION 


The purpose of this paper has becn to appraise the competitive computer 
selection system used by the U.5. Navy. First, the Navy's systcm was defined and 
then alternative mcthodologics were reviewed in order to have a basis for the 
appraisal. 

The Navy's competitive computer selection system is a highly regulated process. 
There are many regulations and policies issued by higher authorities such as the 
@e7eress, the President, the Secretary of Deicnse and others. 

The Navy does however, have a great deal of latitude in deciding exactly which 
methodologies will be used to differentiate among vendor proposals. 

The Automatic Data Processing Equipment Selection Office is the organization 
within the Department of the Navy charged with the responsibility for the development 
and administration of the Department of the Navy plans, policies, procedures, and 
methods governing ADPE specification preparation, proposal solicitation, proposal 
evaluation and selection. 


The ADPESO is located in the office of the Special Assistant to the Secretary of 


the Navy, who is designated as the scnior ADP policy official in the Navy. 
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The Navy's competitive computer selection process consists of five major 

activities: 
Specification Review 
Preparation of the RFP and Selection Plan 
Vendor Liaison and Validation of Proposals 
Proposal Evaluation 
Contract Negotiation and Award 

The timing of the proposed computer configurations is validated by using 
benchmark programs. Jn conjunction with these programs, a probabilistic workload 
projection technique is used to define the monthly cost to perform the workload. ‘The 
monthly costs are consolidated into yearly increments to arrive at the annual costs 
which in turn are discounted to represent the present value of the total cost to 
perform the workload over the total system life. 

The Cost- Value Technique is used to evaluate the desirable features offered by 
the vendors. The distinguishing feature of this technique js the assignment of value 
to desirable features in terms of dollars. Thus, an important benefit derived from 
the use of this method is improved management understanding of the evaluation 
process, thereby facilitating meaningful decisions relative to the value of any or all 
of the desirable features. 

By contrast, the Navy could validate the timing of the proposed systems by 
calculating the times manually or by using a computer simulation routine such as 


SCERT. These methods however do not provide data that is as reliable as the bench- 


mark programs. In addition, a cost study conducted by the U.S. Army found that 
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benchmark programs could be prepared and used at a lower cost than the other two 
techniques. 

The Cost- Effectiveness and Weighted Scoring systems arc alternative processes 
that can be used to evaluate vendor proposals. The Cost-lEffectiveness and Cost- 


Value systems are very similar. There is a slight difference in assigning worth to 





desirable features. There is a marked difference between the Cost- Value and the 
Weighted Scoring techniques. The most significant is that Cost- Value uses dollars 
as an evaluation scheme, .while the laiter awards points based on a predefined point 
scoring templet. These scores are added up, and the system with the most points 
ostensibly is the best choice. 

The flaw in Weighted Scoring system is that the factors to be evaluated and the 
weights assigned to them are by necessity, chosen arbitrarily. There are no 
objective guidelines for matching factors and weights to a particular users necds. 

The Author has made one recommendation to improve the Navy system. It has 
been suggested that a methodology advocated by David Hertz, called Risk Analysis 


be incorporated in the system. 
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LIN Public Law 89-306 
Smee 89th Congress, H. R. 4845 
wN Cy October 30, 1955 


an See 


‘eo provide for the economic and e@iclent purchase, lease, mainteaanece, opera- 
tion, and utilization of automatic data processing equipment by Federal 
departments and asencies, 


79 STAT. 1227. 


Be tt enacted by the Senate and Mouse of Representatives of the 
United States of America in Congress assembled, That title 1 of the Automatio data 
~ e e . ’ ° ‘ es +. 
Federal Property and Administrative Services Act of 1949 (63 Stat. processing 


377), 25 amended, is hereby amended by adding a new section to read equipment. 
as follows: Purchesa and 


‘ ubilizatcicn. 
“AUTOMATIC DATA PROCESSUNG EQUIPMENT 5 uSC 630--639—-1 


“Sec. 111. (a) The Administrator is authorized and directed to 
coordinate xnd provide for the economic and eflicient purchase, lease, 
and maintenance of automatic data processing equipment by Federal 
POHCIES. | 

“(b) (1) Automatic dats processing equipment suitable for eficient 
and cffective use by Federal agencics shal! be provided by the Adminis- 
trator through purchase, lease, transfer of equipment from other 
Federal egencies, or otherwise, and the Adininistretor Js authcrize 
and directed to provides by contract or otherwiss for the maintenance 
and repair of such equipment. In carrying out his responsibitities 
under this section the Administrator is authorized to transfer auto- 
matic data processing equipment between Federal agencies, to provides 
for joint utilization of such equipment by two or more Federal 
agencies, and to establish and cperate equipment pools and dats 
processing centers for the use of two or more snch agencies when 
necessary for its most efficient and effective utilization. 

“(2) The Administrator may delegate to one or more Federal 
agencies authority to operate automatic data processing equipment 
pools and automatic data processing centers, and to lease, purchase, 
or maintain individual automatic data processing systems or specific 
units of equipment, inchiding such equipment used in automatic data 
processing pools and automatic dsta processing centers, when such 
action 1s determined by the Administrator to be necessary for the 
economy and eficiency of operations, or when such action is cssential 
to national defense or national security. The Administrator may 
delegate to one or inore Federal agencics authority to lease, purchase, 
or maintain automatic data processing equiprnent to the exleni to 
which he determines such action to be necessary and destrable to sllow 
for the orderly implementation of 2 program for the utilization of 
such equipment. 

“(c) There is hereby authorized to be established on the books ADP fund, estabd- 
of the Treasury an antomatic data processing fund, which shall be lishment. 
available without fiscal year limitation for expenses, including per- 
sonal services, other costs, and the precurement by lease, purchase, 
transfer, or otherwise of equipment, maintenance, and repair of such 
equipment by contract or otherwise, necessury for the eficient coordl- 
nation, operation, utilizetion of such equipment by end for J*ederal 
agencies: Provided, That a report of equipment inventory, utiliza- rquipment aoqui- 
tion, and acquisitions, together with an account of receipts, disburse- sitions end util 
ments, and transfers to misccllaneous receipts, under this authoriza- ization. 


85-081 O - 67 
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Pub. Law 89-306 Be October 30, 1965 


~ 


79 STAY. 1328 


Report to 
Budget bureau 
end Congress. 


Appropriation. 


63 Stat, 384. 
40 USC 481, 474. 


Scientific and 
technological 
advisory serv- 
ices by Seore- 
tary of Conmerce. 


tion shali be made annually in connection with the budget estimetes to 
the Director of the Bureau of the Budget and to the Congress, and 
the inclusion in 2ppropriation acts of provisions regulating the opera- 
tion of the automatic data processing fund, or limiting the expendi- 
tures therefrom, is hergby authorized. 

“(d) There are authorized to be appropriated to seid furid such 
sums as nay be required which, together with the value, as deternined 
by the Administrator, of supplies and equipment from time to tine 
transferred to the Administrator, shall constitute the capital of the 
funda: Provided, Thet. said fund shall be credited with (1) advances 
aud reimbursements from available appropriations end funds of any 
agency (including the General Services Administration), organiza- 
lion, or contractor utilizing such equipment and services rendered 
them, at rates determined by the Administrator to approximate the 
costs thereof met by the fund (inchiding depreciation of equipment, 
provision for accrued Jeave, and for amortization of installation costs, 
but excluding, in the determination of rates prior to the fiscal year 
1967, such direct operating expenses as may be directly appropriated 
for, which expenses may be charged to the fund and covered by 
advances or reimbursements from such direct. appropriations) and 
(2) refunds or recoveries resulting from operstions of the fund, 
niciuding the net proceeds of disposal of excess or surplus personal 
property aud receipts from carriers and others for less of or damage 
to property: Provided further, That following the cloce of each fiscal 
year aly net meome, after making provisions for prior year losses, 
if any, shal] be transferred to the Treasury of the Uniled States as 
miscellaneous receipis. 

“(e) The proviso following paragraph (4) in section 201(a) of this 
Act and the provisions of section 602(d) of this Act shal] have no ap- 
plication in the administration of this section. No other provision 
of this Act or any other Act whieli is inconsistent with the provisions 
of this section shall ‘be applicable in the administration of this section. 

“(f) The Secretary of Commerce is authorized (J) to provide 
agencies, and the Administrator of General Services in the exercise 
of the authority delegated in this section, with scientific and techno- 
Jogica] advisury services relating to automatic data processing and 
related systems, and (2) to make appropriate recommendations to 
the President. relating to the establishment. of uniform Tederal auto- 
matic data processing standards. The Secretary of Coininerce 15 
authorized to undertake the necessary research in the sciences and 
technologies of automatic data processing compnter and related sys- 
tems, as may be required under provisions of this subsection. 

“(¢) The authority conferred upon the Adinimstretor and the 
Secretary of Commerce by this scetion shall be exercised subject to 
direction by the President and to fiscal and policy control exercised 
by the Bureau of the Budget. Authority so conferred upon the 
Administrator shall not be so construed as to impair or interfere with 
the determination by agencies of their individual automatic data proc- 
essing equipment requirements, including the development of specifi- 
cations for and the scleetion of the types and configurations of equip- 
ment needed. ‘Sie Administrator shall not interfere with, or attempt 
to control in any way, the use made of automatic data processing 
equipment er components thereof by any agency. The Administra- 
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October 30, 1965 =.= Pub. Law 89-2306 
79 STAT, 1129 


tor shall provide adequate notice to all agencies and other users con- Notice to 
cerned with respect to each proposed determination specifically affect- agencies. 
ing them or the automatic cata processing equipment or components 

used by them. In the absence of mutual agreement. between the 
Administrator and the agency or user concerned, such proposed deter- 
ninations shall be subject to review and decision by the Bureau of the 

Budget unless the President otherwise directs.” 


Approved October 30, 1965. 
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Use 


Hl a 


and Management of Co omputer 


The Presiden 


’s Afenoranduan to Heads of Departinents 
end A .. S. 


frne 28, L966 


J want the head of every Federal agency to explore and 
apply all possible incans to 

-—use the electronic computer to do a better job 

——Inalage computer activity at the lowest possible cost. 

Iwant my admunotration to give prioiity empuasis to 
both of these objectives—nothing less will suffice. 

The electronic computer js having a greater impact on 
what the Government does and how it does it than any 
other product of modern technology. 

The computer ts inaking it possible to 

—send men and satellites into space 

-——~make significant strides in medical research 

—add several billions of dollars to our revenue through 
nnproved tax adiniuistration 

——admunister the huge and complex social security and 

medicare programs 

-—-manage a multi-billion dolar defense logistics system 

—speed the issuance of G.I. insurance dividends, at 

much: Jess cost 

—-save lives through better search: and rescue operations 

—harness atomic energy for peaceful uses 

—design better but less costly highways and structures. 

In short, computers are enabling us to aclueve progress 
and benefits which a decade ago were beyond our grasp. 

The technology is avilable. Hts potential for good 
has been amply demonstrated, but it remains to be tapped 
in fuller measure. 

Iam determined that we take advantage of this tech- 
nology by using it maginatively to accomplish worthwhile 
purposes. 

I therefore want every agency lead to give 
Stacy to new ways in which the electronic computer 
might be uscd to 

provide better service 

-— Iniprove 


thorough 


to the public 
sagency performance 
—reduce costs, 
But, as we use computers to achieve these benefits, | 
want these activities managed at the lowest posatble cost. 
At the present time, the Federal Government 
—uses 2,600 computers | 
— employs 71,000 people in this activity 
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—speuds over $2 billion annually to acquire and op- 
erate this equipment, including special niuitary type 
cOMpUlCTs 

Clearly, we nist devote our best efforts to managing 
this large investment wiscly and with the least cost. 

I approved a blueprint for action when T approved the 
Bureau of the Budget “Report on Management of ADP 
in the Government.” , 

Phe Congress recognized this need when it enacted 
Public Law 89 306 (the Brooks Bill) last Octcber. “This 
legislation provided specific authorities to 

—the General Services Administration, for the pro- 
curement, utilization and disposition of automatic 
data processing eqanpinent, 

—-the Department of Gommierce, for the development 
of data processing standards anal the provision cf 
assistance to agencies m designing computcr-based 
Sysfcims 

—the Bureau of the Budget, for cxerci-ing policy and 
fiscal control over the implementation of these 
authoritics. 

These agencies are secking actively to put into effect 
ways for improving and reducing the cost of this huge and 
complex operation. 

In my Budect Message for 1967 I told the Congress 
of my intent to make sure that this huge investment 1s 
managed efficiently. 

The Federal Goveriment must give priority attention 
to 

-——-establishing better and more effective procurement 

methods 
—miaking fuller use of cxtdug facihties Uirougl: shar- 

ing aud joint-use arrangements before acquiring 
additional equipment 
ge excess equipment whenever feasible 
—achieving, with lidustry cooperation, greater com- 
patibility of equipment. 


Uti zine 


T expect all agencies to cooperate fully with the Bureau 
of the Budget, the General Serviccs Administration, and 
the Department of Commerce tn accomplishing these 
objectives. 

J want the Director of the Burcau of the Budget to 
report tome on December 31, 1966, and every stx months 
thereafter, on the progress that is being made throughout 
the Federal Government in improving the managenient 
of thus very nnportant technology. 


Lynpvon D3. JorNnson 
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APPENDIX E 


DIRECTIVES GOVERNING ACTIONS OF ADPES 


Bureau of Budget Circulars 


Sec eS 


A-27, Policies and Responsihilities of the Sharing and Electronic Computer 
Diners er tecsin the ExXecullyesbranel 


A-54, Policies on Selection and Acquisition of Automatic Data Processing 
(ADP) Equipment 


A-61, Guidelines for Appraising Agency Practices in the Management of 
Automatic Data Processing (ADP) Equipment in Federal Agencies 


A-94, Discount Rates and Procedures.to Be Used Evaluating Deferred Costs 
and Benefits 


Wepartiment of Defense Instructions 


4160.19, DOD Automatic Data Processing Equipment Reutilization Program, 
May 8, 1969 


7041.3, Economic Analysis of Proposed Department of Defense Investments, 
February 28, 1969 


9100.40, Responsibilities for the Administration of Automatic Data Processing 
Equipment Program, September 28, 1963 


4160.1, Nonexcess Personal Property to be Sold or Exchanged for Replace- 
ment Purposes, August 10, 1966 


4160.21, Department of Defense Personal Property Disposal Program, 
April 28, 1967 
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© 4140.34, Department of Defense Personal Property Utilization Program, 
September 5, i968 


Department of the Navy Instructions 


a) Comptroller of the Navy Instruction 7000.28, Economic Analysis of Proposed 
Department of the Navy Investments, April 19, 1967 


© secretary of the Navy Instruction 10462.7B, Department of the Navy Automatic 
Data Processing Program, March 11, 1966 





APPEND a 
AUTOMATIG DATA PROCESSING SYSTEM SPE CIVICATIONS 
CONTENTS 


Part I - Supporting Information 
Section 1 - Justification for Acquisition of ADP System 
l- Ll sions, 
]-2 SY Stemi istificartton 
1-3 Budgeting Estimates and Limitations 
Section 2 Vortec esenus ion 
2-1 Representative Programs 
2-2 Derivation of Representative Programs 
section 3 - Explanation of Requirements 
3-1 Justification of Mandatory Requirements 
Oo 2 ottemiems Of Weaimable PCarnres 
Op om ertecmmCcnie Of Vale 


ert i! - ADP System Specifications 
Section 1 - Format and Content of ADP System Specifications 


Pa ACINEENT 3 


Chapter 1 - Background and Operational Plans 
1-1 Background 
1-2 Operational Plans 
Chapter 2 - Requirements 
2Z-l oy siem Mamdatory Kequirenrents 
2-2 Desirable Features 
Chapter 3 - Explanation of Terms and Abbreviations 


ATTACHMENT II 


Chapter 1 - System Questionnaire 
Chapter 2 - Timing Information 
2-1 General Information 
2-2 Prorram Weseriprion 
2-3 Extension Factors and Timing Tables 
2-4 Benchmark Demonstration Instructions 
Section 2 - Benchmark Package 


SOURCE: U.S., Department of the Navy, Specification, selection, and 


Acquisition of Automatic Data Processing Equipment, Secretary of the Navy. 
Instruction 10462.13, Department of the Navy, Washington, D.C., 20390, p. B-2. 
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EMR 


6000 Series 


Burroughs 
HOO Series 
ZOU Serics 
300 Series 


2500 
3500 
5500 
6500 


Control Data 


160A 


1604A 


1700 
3100 
3200 
3300 
3400 
3500 
3600 
3800 
6400 
6500 
6600 


General Electric 


es 
120 
25 
215 
225 
235 
415 
425 
435 
625 
635 


APPENDIX G 
COMIPUTER SYSTEMS IN SCERT LIBRARY 


Honeywell 
110 
10 
2s 
200 
400 
800 
1200 
1250 
1400 
1800 
2200 
3200 
4200 
8200 


IBM 
360/20 
360/25 
360/30 
360/40 
360/44 
360/50 
360/65 
360/67 
360/75 
360/85 
360/195 
PSO 

140] 

1410 

1440 

1oZ0 

SONG. 

7040 Series 
Os 

7070 Series 
7080 

7090 Series 


NCR 

oe 

Shek MC 

GEN TURY-160 
CENTURY-200 
Tete 

1900 Series 


System 4 Series 
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Divital Lquipiment 
PIDP=<6 
PDP- & 
PDP- 81 
PIDPsiO 


Phaieo 
2000 Series 


RCA 
30] 
S01 
601] 
S518) 
70/15 
70,25 
70/35 
70/45 
70/46 
70/55 
70/60 


UNIVAC 
I] 

418 
418-1] 
490 Series 
1004 Series 
1005 
1050 
1106 
iO? 
1108 
9°00 
9300 
9400 


SIS 
920 
O25 
930 
9300 
io taae 
Sigma 5 
Sic” 





APPEND Pan 


PRODUCTION OF OUIZUTREEGhi.> 


SCERT has been designed to produce a complete scries of "standard" output 
reports but also to be readily adaptable to the production of nonstandard or special 
purpose reports. The reports of the standard sct are all programmed as optional, 
and the typical use of the program calls for just those reports needed for one's 
particular analysis. A brief description of the standard reports: 


Computer Complement Report - Portrays the exact configuration that was 
simulated and provides certain basic cost data about that configuration. 


Central Processor Utilization - Summarizes for each scheduled run the 
projected running time, set-up time, and horizontal memory utilization. 


: Programming Requirements - Projects for each run an implementation- oriented 
by-product of the simulation by summarizing the number of steps to be 
programmed and aun estimate of the number of programmer man months 
required. 


Application Summary - If the processing requirements simulated can be 
broken into discreet application areas, this report summarizes running 
time and programming effort for each of these areas. 


Computer Capabilities - Tabulates for cach scheduled run simulated break- 
out of thru-put requirements and then a total for the entire simulation. It is 


used primarily to pinpoint critical hardware areas. 


Cost Summary - Primary cost output which relates the projected utilization of 
the computer to lease, purchase, and maintenance cost considerations. 


lO? 





108 


Real-time Analysis - If random real-time processing was involved, a serics of 
reports are produced to portray the impact of such processing on the computer 
and the response expected of the coinputer for such processing. fhe ion: 
standard reports are: 


Event Processing Analysis - Analyzes each random event in terms of 
its unique thru- put. 


Hardware Utilization - Analyzes every potential queue present in the 
hardware complex. 


Syvetems Mesponuse - Retlécts the expecred 95ti anceyo isp encemmleno: 
computer and communications network response. 


Memory Requirements = Outlines the unique memory neqmine asiemese 
random event and the expected and worst case background requirements. 


Multiprogram Run Schedule - Produced whenever multiprogramming has been 
simulated and reflects those runs which have been scheduled by the simulator to 
operate concurrently. | 


Detail Analysis - Produced for each schedtiled run or veal-tine evene cimulaited 
and serves as backup for all other reports. It precisely portrays the utilization, 
timing, and memory requirements derived by the simulation and presimulation 
algorithms for all components making up the run. 


Summary Analysis - Summarizes, for each configuration simulated, the 
performance and cost/performance derived. 


System Documentation - A by-product report which produces a standard form of 
documentation of all processing requirements. It is especially useful in securing 
responsive, realistic hardware bids from computer manufacturers. 





APPENDIX J 


SAMPLE FORMAT FOR DISCOUNTING DEFERRED COSTS AND BENEFIT?S 


Assume a tensyvearc program which walecoumiupeeneomeo nines 
the stream of expenditures appearing in column (2) of the 
table below amd which will Pesuie tne sericea en bone die 
appearing in column (3). The discount factor for a 10 percent 
aiscount rate is presented in column (4). Present value cost 
zOr Cach ef eke ten yeers 15 calleulaced) by muerte emae 

column (2) by column (4); present value benefit for each of 
ehe cengvears 16 calculatedsoy7 mule lyin mco Lommemcsy mee 


column (4). Present Nagshs costs and benefits are presented 
in columns (so) cand (6), respegeively. 
PoesSCie Presence 
Value Value 
baepected EXpeeecd. | paseoun . Cost Benefit 


Year of Yearly Yearly. Factor tor (Gola) eel cotm ec. 
Cpe ra clone cost Benet iteel 0 Percent “Colm eee ame) 


(ay) (2) (3) (4) Cs) (6) 





1 $10 San OS] (0)9 Soa $ 0.0 

2 28 0 O=626 LGe 0.0 

S 30 5 Git ol 225 See) 

4 30 1G GewioS 20k Ge 

5 20 30 oe 2.) U2 hE 36 

6 ia £0 9.564 Sie 6 2256 

7 5 40 Opes te Za VAS, 

8 5 £0 0.467 2S IS ey, 

S) 5 £0 0.424 2a tse 

10 5 ZS C= 2.0 la, Shad 
SSeS: S Asdigiaeae 

ThessuneOn Co@lumi (5) 1S present value cose: “S9o.2: 


The sum of column (6) is present value benefit: $117.7. 


Present value net benefit is the difference between present 
value total benefit and present value total cost: 
Sei) = opp O22. 


Thies bene tit -cectererio 1S Vl 77 Joes eee 


NOTE: For More Gitfticult discounting vreblcns, = Fecoms 
mended reference 1S Principles Of Pics Sang 
Economy, by Eugene ae Grant TAT ie Ger re CON, 
Ronald Press Company, 1960. 
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